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FOREWORD 


The  Air  Force  plans  to  publish  annual  reports  demonstrating  the  utilization 
of  people-related  research,  development,  test,  and  evaluation.  Since  this  is 
the  first  such  report,  it  includes  significant  research  from  fiscal  years  1980 
through  1982.  This  report  was  prepared  jointly  by  personnel  from  the  Air 
Force  Manpower  and  Personnel  Center,  and  the  Applications  and  Liaison  Office 
of  the  Air  Force  Human  Resources  Laboratory.  Utilization  of  research  products 
in  the  people- related  areas  of  manpower  and  personnel,  human  factors,  educa¬ 
tion  and  training,  and  simulation  and  training  devices  are  described.  The 
products  of  large-scale,  multi-year  programs,  as  well  as  of  smaller  project 
efforts  in  key  areas,  are  included. 

This  report  does  not  encompass  the  total  spectrum  of  activities  of  the 
people- related  research  program  of  the  Air  Force.  The  examples  presented  here 
show  some  of  the  more  significant  and  interesting  cases  or  research  utiliza¬ 
tion  and  of  technology  base  advancement.  They  are,  however,  only  a  small 
representative  sample  of  the  people-related  research  performed  continually  by 
the  Air  Force.  The  primary  emphasis  in  this  report  has  been  on  utilization  by 
Air  Force  operational  communities  and  on  the  benefits  in  both  cost  and  opera¬ 
tional  effectiveness  resulting  therefrom.  Special  attention  has  been  focused 
on  the  users'  perspectives.  This  report  contains  (a)  examples  of  research 
results  already  in  use,  (b)  advances  in  the  technology  base  for  near  future 
use,  and  (c)  current  activities  designed  to  increase  such  utilization. 

This  report  has  been  prepared  primarily  for  policy  decision  makers  within 
the  Department  of  Defense  (DoD),  legislative  personnel  of  appropriate  Congres¬ 
sional  committees,  and  users  or  sponsors  from  Air  Force  operational  and 
training  commands.  Funding  for  people- related  research  and  development  is 
provided  on  the  basis  that  the  results  will  significantly  impact  the  opera¬ 
tional  readiness  of  Air  Force  units  by  providing  the  technology  for  efficient 
management  and  utilization  of  Air  Force  personnel  and  by  improving  the  per¬ 
formance  of  personnel  and  Air  Force  weapon  systems.  This  assumption  of 
operational  utility  is  increasingly  requiring  validation  because  justifica¬ 
tions  for  funding  requests  are  more  carefully  scrutinized.  This  report  is  one 
such  validation  effort  prepared  in  response  to  a  joint  directive  from  Head¬ 
quarters,  United  States  Air  Fqrce/Deputy  Chief  of  Staff,  Manpower  and 
Personnel,  and  Headquarters,  Air  Force  Systems  Command. 
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INTRODUCTION 


This  report  describes  recent  Air  Force  "people-related"  research,  develop¬ 
ment,  test,  and  evaluation  efforts  that  have  been  utilized  by  operational 
commands  or  have  contributed  to  the  technology  base  that  supports  present  and 
future  operational  needs  of  the  Air  Force.  People- related  research  and 
development  (RSD)  is  concerned  with  issues  of  manpower,  personnel,  and 
training.  Its  purpose  is  to  enhance  operational  readiness  by  improving  the 
attitudes,  performance,  and  management  of  Air  Force  personnel. 

People-related  research  is  funded  under  the  Training  and  Personnel  Systems 
Technology  program  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Research  and  Engineering.  This  program  has  four  major  objectives,  which  also 
provide  the  organizational  framework  for  this  report.  These  objectives  are: 

1.  Manpower  and  Personnel.  To  improve  the  ability  of  the  Air  Force  to 

determine  both  quantitative  and  qualitative  needs  for  people  and  to  recruit 
and  retain  personnel  in  sufficient  numbers  based  on  current  and  future  mission 
requirements. 

2.  Human  Factors.  To  improve  the  design  of  weapon  and  support  systems  so 

that  a  wide  range  of  personnel  can  operate  and  maintain  equipment  effec¬ 

tively.  When  well  done,  human  factors  R&D  minimizes  workloads,  skill  levels, 
and  training  requirements  associated  with  Air  Force  systems. 

3.  Education  and  Training.  To  develop  knowledges,  skills,  and  attitudes 
in  military  personnel  essential  to  effective  performance  of  duty  assignments 
throughout  their  careers. 

A.  Simulation  and  Training  Devices.  To  improve  the  effectiveness  of 

military  training  at  reduced  cost. 

The  Need  for  People-Related  Research  and  Development 

Today,  the  Air  Force  is  equipped  with  the  most  advanced  and  complex  weapon 
systems  ever  devised.  Well- trained  and  highly-motivated  people  operate  and 
maintain  these  systems.  Yet,  a  look  into  the  80' s  and  beyond  shows  develop¬ 
ments  that  are  at  once  exciting  and  promising,  yet  disquieting  and  chal¬ 
lenging,  and  these  appear  to  dominate  the  horizon.  People  face  monumental 
changes  in  the  way  they  work,  handle  information,  and  fight  wars.  The  indus¬ 
trial  revolution  is  being  replaced  by  the  information  age.  Travel  beyond  the 
earth's  atmosphere  is  a  reality.  Every  aspect  of  air  power  is  changing 
rapidly.  The  events  and  conditions  of  the  wars  that  threaten  the  United 
States  now,  and  of  those  that  may  have  to  be  fought  in  the  future,  forge  an 
inseparable  link  between  the  airman  and  the  scientist,  the  historian  and  the 
futurist. 

Military  training  is  a  major  activity  of  the  Department  of  Defense  (DoD). 
On  an  average  day  in  Fiscal  Year  1982  (FY82),  about  205,000  active  duty  per¬ 
sonnel  and  about  33,000  National  Guardsmen  and  Reservists  underwent  some  type 
of  formal  training  in  one  or  more  of  nearly  9,000  organized  courses  of  in¬ 
struction  developed  and  maintained  by  the  military  training  organizations. 
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The  cost  of  this  effort  was  about  $9.5  billion  in  FY82,  and  the  support  of 
about  92,000  military  and  civilian  personnel  was  reauired  for  formal  instruc¬ 
tion,  instruction  support,  school  administration,  and  student  supervision. 
About  1.8  million  officer  and  enlisted  active  duty  personnel  benefited  from 
this  activity  in  FY82.  Every  index  indicates  a  diminishing  ratio  of  training 
resources  to  training  requirements.  The  solution  to  this  dilemma  is  to  in¬ 
crease  training  productivity  through  application  of  available  and  emerginq 
high  technology.  It  is  essential  that  effective  personnel  and  training 
research  programs  be  maintained  to  supply  timely  information  to  decision 
makers  in  these  areas. 

The  Importance  of  People-Related  Research  and  Development 

Although  the  military  Services  are  fielding  technically  superior  weapon 
systems,  achieving  desigped-in  performance  requires  that  these  systems  be 
adequately  manned,  maintained,  and  supported.  Otherwise,  technologically 
advanced  aircraft,  ships,  ordnance,  and  other  systems  will  not  perform  as 
designed.  They  cannot  do  so  without  adequate  attention  to  the  reliability  of 
the  system's  human  component.  This  can  be  accomplished  only  through  a 
continuing  interface  between  system  designers  and  manpower  planners  whose  man¬ 
power  requirements  influence  system  design.  People-related  research  and 
development  can  supply  much  of  the  information  needed  by  system  planners  and 
designers. 

The  issues  of  recruitment,  attrition  and  retention,  training,  morale,  and 
readiness  are  all  closely  interlinked.  High  attrition,  for  example,  means 
higher  recruitment  needs,  increased  training  costs,  and  lower  morale.  Conse¬ 
quently,  the  research  efforts  reported  here  often  have  secondary  impacts  in  a 
variety  of  areas  other  than  their  immediate  focus. 

There  is  growing  evidence  that  the  issue  of  manpower  availability  and 
trainability  may  be  one  of  the  most,  if  not  the  most,  limiting  factor  on  the 
amount  of  Air  Force  truly  available  in  the  future.  In  the  words  of  George  S. 
Patton,  Jr.,  "Wars  may  be  fought  with  weapons,  but  they  are  won  by  men.  It  is 
the  spirit  of  the  men  who  follow  and  the  man  who  leads  that  gains  the  victory.” 
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MANPOWER  AND  PERSONNEL 


The  Department  of  Defense  defines  the  manpower  and  personnel  area  of 
people- related  research,  development,  test,  and  evaluation  as  follows: 

"Development  of  techniques/methods  for  utilizing  available  personnel 
resources  through  improved  selection,  job  assignment,  organizational  analysis 
and  management  techniques  to  meet  combat-available  and  projected  force  needs." 

The  Air  Force  requires  a  continuing  supply  of  quality  men  and  women  who 
can  operate  and  maintain  sophisticated  weapons  and  support  systems.  The  iden¬ 
tification,  acquisition,  and  retention  of  a  quality  manpower  pool  is  diffi¬ 
cult.  The  proportion  of  quality  men  and  women  who  are  adaptable  to  military 
life  is  limited  and  their  recruitment  is  subject  to  fierce  competition  from 
the  civilian  sector.  The  number  of  individuals  of  suitable  age  is  declining 
now;  therefore,  it  is  even  more  difficult  to  attract  individuals  to  the  Air 
Force.  It  is  incumbent  upon  the  Air  Force  to  ensure  that  every  effort  is  made 
to  identify  individuals  who  might  contribute  to  the  Air  Force  mission,  to 
assign  them  to  jobs  that  match  as  nearly  as  possible  their  talents  and  aspira¬ 
tions,  and  to  retain  them  in  productive  Air  Force  jobs  as  long  as  they  are 
needed. 

Projects  in  this  category  include: 

1.  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 

2.  Enlisted  Person- Job  Match  (PJM) 

3.  Aptitude  Requirements  for  Air  Force  Jobs 

A.  First-Term  Enlisted  Force  Cost  Study 

5.  Retraining  and  Transferability  of  Skills 

6.  Air  Force  Reading  Abilities  Test  (AFRAT) 

7.  Computerized  Adaptive  Testing  (CAT) 

8.  Organizational  Assessment  Package  (OAP) 


ARMED  SERVICES  VOCATIONAL  APTITUDE  BATTERY 


Need 


The  most  effective  use  of  available  personnel  is  the  basic  goal  of  every 
organization.  When  individuals  are  placed  on  jobs  for  which  they  are  not 
prepared,  the  result  is  frustration  for  the  individual,  expense  to  the  organi¬ 
zation,  and  a  waste  of  manpower.  Conseguently ,  over  the  past  70  years, 
psychological  testing  has  come  into  increasing  use  as  a  guide  to  improve 
personnel  administration.  Testing  is  used  to  lower  training  costs,  to  redjce 
risks  in  hazardous  occupations,  to  optimize  the  allocation  of  scarce  re¬ 
sources,  to  enhance  retention  of  skilled  and  experienced  personnel,  and  to 
increase  on-the-job  productivity. 

Testing  programs  have  been  applied  to  two  personnel-oriented  tasks.  The 
first  is  personnel  selection,  wherein  the  best  gualified  job  applicants  are 
chosen  from  those  available.  The  second  task  is  the  more  complex  one  of 
classification;  an  attempt  is  made  to  assign  all  selected  applicants  on  the 
basis  of  personal  abilities  to  the  jobs  in  which  they  can  be  most  effective. 
The  ultimate  goal  for  both  procedures  is  the  same:  to  maximize  the  overall 
effectiveness  of  the  total  organization. 

From  1948  to  1975,  the  Air  Force  used  multiple  aptitude  batteries  to 
classify  nonprior-service  enlisted  personnel.  During  this  time,  10  different 
operational  batteries  were  used.  In  early  1966,  the  Assistant  Secretary  of 
Defense  for  Manpower  and  Reserve  Affairs  directed  the  military  Services  to 
explore  the  feasibility  of  creating  a  common  aptitude  test  battery.  A  working 
group  consisting  of  personnel  testing  experts  from  each  of  the  military 
Services  was  set  up  to  study  the  feasibility  of  such  a  test  battery  and  to 
develop  a  prototype.  The  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 
was  developed  from  this  effort. 

By  direction  of  the  Assistant  Secretary  of  Defense  (Manpower  and  Reserve 
Affairs),  effective  1  January  1976,  the  Army  became  Executive  Agent  for  the 
centralized  management  of  ASVAB  testing.  As  such,  the  Army  assumed  responsi¬ 
bility  for  managing  ASVAB  as  the  common  DoD  enlisted  accessions  test,  both  at 
■alitary  Entrance  Processing  Stations  and  in  the  DoD  High  School  Testina 
Program.  The  Air  Force,  however,  continued  as  Executive  Agent  for  ASVAB  R&D 
with  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  designated  as  the  lead 
DoD  laboratory.  Thus,  there  exists  an  on-going  reguirement  for  AFFRL  to 
develop  new  versions  of  ASVA8,  to  validate  and  standardize  them,  and  to 
perform  research  and  development  (R&D)  for  ASVAB  program  improvement. 

R&D  Solution 


This  project  was  conducted  by  the  Manpower  and  Personnel  Division  of 
AFHRL.  It  was  performed  in  response  to  a  reguest  from  the  Personnel  Testing 
Branch  of  the  Air  Force  Manpower  and  Personnel  Center  as  well  as  in  response 
to  a  DoD  directive.  The  approach  taken  in  this  project  was  initially  a 
tri-Service  effort.  The  Army,  through  its  Behavior  and  Systems  Research 
Laboratory  (now  the  Army  Research  Institute) ,  was  the  lead  Service  in  an 
accelerated  program  to  determine  to  what  extent  aptitude  tests  of  the  several 
Services  were  interchangeable.  Test  and  measurement  personnel  from  AFHRL  and 
the  Navy  Personnel  Research  and  Development  Center  collaborated  with  the  Army 
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in  the  development  of  the  first  ASVAB.  The  objective  of  the  study  was  t-, 
identify  which  of  the  classification  tests  of  the  Army,  Navy,  and  Air  Force 
were  interchangeable  in  terms  of  abilities  and  aptitudes  measured,  and  fr^r 
those  so  identified,  to  develop  shortened  forms  to  constitute  an  alternate 
inter-Service  battery  that  would  not  require  testing  time  in  excess  of  2-1/? 
hours.  Comparability  of  the  several  Service  tests  was  determined  from  inter¬ 
correlations  on  a  consolidated  enlisted  sample.  This  sample  was  stratified  on 
the  Armed  Forces  Qualification  Test  (AFQT)  to  provide  a  representative  distri¬ 
bution  of  ability.  The  Army,  Navy,  and  Air  Force  each  provided  1,000  sub¬ 
jects,  and  the  Marine  Corps  provided  300. 

Seven  sets  of  tests  were  identified  as  interchangeable:  word  knowledge, 
arithmetic  reasoning,  space  perception,  mechanical  comprehension,  shop  in¬ 
formation,  automotive  information,  and  electronics  information.  The  Army 
Coding  Speed  Test  was  selected  as  a  measure  of  clerical  aptitude  on  the  basis 
of  separate  validity  studies. 

It  was  necessary  to  abbreviate  the  tests  identified  as  interchanaeable  so 
that  the  new  battery  would  not  exceed  the  time  limit  imposed.  To  this  end,  25 
items  were  selected  from  each  set  of  three  interchangeable  tests  to  provide 
one  test  about  half  the  length  of  each  parent  test.  Items  with  a  wide  range 
of  difficulty,  from  very  easy  to  very  difficult,  were  selected  so  that  each 
new  test  could  measure  a  wide  range  of  ability.  After  editing,  the  selected 
items  were  organized  into  a  battery  of  nine  tests,  the  ASVAB.  The  new  battery 
was  standardized  on  a  sample  of  3,000  selective  service  registrants,  again 
stratified  on  Al'TT. 

Since  its  original  development,  ASVAB  has  been  revised  on  a  2-year  cycle 
with  no  break  in  effort.  These  revisions  require  repeated  standardization  and 
validation  against  Air  Force  and  civilian  criteria.  Furthermore,  ASVAB  is 
revised  periodically  to  offset  test  compromise;  thus,  this  is  a  continuous 
program. 


Application 

The  ASVAB  has  been  utilized  for  the  following  purposes:  (a)  more  effi¬ 
cient  military  personnel  selection  and  classification,  (b)  improved  service 
recruiting  efforts,  and  (c)  enhanced  usefulness  as  a  quidance  device  for  hiah 
school  counselors. 

The  ASVAB  has  been  extensively  modified  to  provide  a  single  battery  which 
is  utilized  by  all  the  Armed  Services.  AFHRL  has  conducted  a  series  of  vali¬ 
dation  studies  to  ensure  that  this  battery  correctly  screens  and  classifies 
enlistees.  Typically,  the  ASVAB  has  been  correlated  with  technical  school 
grades.  Validity  coefficients  against  school  grades  were  computed  within  each 
of  119  technical  school  courses  in  the  Air  Force.  Within  the  aptitude  areas, 
i.e.,  Mechanical  (M ) ,  Administrative  (A),  General  (G),  and  Electronics  (E), 
the  following  ranges  and  median  uncorrected  validities  occurred: 


M  A 

G 

E 

No.  Of 

Courses  28  13 

33 

45 

Range 

.09-. 69  .03-.  42 

.23-. 61 

.13-. 67 

Median 

.35  .24 

.43 

.39 

These  validity 

coefficients  indicate 

that  the 

ASVAB  is  predictive, 

moderate  degree,  of  Air  Force  technical  school  achievement. 


5 


Another  way  of  determining  the  impact  of  ASVAB  is  to  estimate  cost  avoid¬ 
ance  that  occurs  through  its  use.  Demonstrating  costs  avoided  throuuh 

application  of  selection  tests  is  not  a  simple  matter.  If  a  researcher  could 

go  out  on  the  street  and  randomly  collect  young  people  and  place  them  in  an 
Air  Force  Technical  School  course,  some  would  succeed,  and  some  would  not.  In 
the  ASVAB  validation  studies,  a  broad  unrestricted  street  sample  does  not 
exist.  There  are  no  direct  data  on  failure  rates  in  the  total  sample;  the 
only  data  pertain  to  the  persons  selected  for  training.  However,  the  perform¬ 
ance  of  the  full  sample  can  be  projected  by  oaliaMflfcBaei* means. 

The  following  conservative  estimates  were  used  to  compute  annual  cost 
avoidance:  an  average  selection  level  of  50th  percentile,  a  corrected  valid¬ 
ity  of  .5,  a  training  attrition  rate  of  5  percent,  and  an  average  eliminee 

training  cost  of  $5,000. 

Based  on  a  normal  distribution,  the  attrition  rate  for  an  unrestricted 
sample  would  be  1A  percent.  To  obtain  the  same  number  of  graduates,  an  addi¬ 
tional  7,326  airmen  would  be  needed  at  an  additional  total  training  cost  of 
$36,630,000  per  year.  Considering  that  the  Air  Force  represents  only  20 
percent  of  the  total  Armed  Services  personnel,  this  estimation  can  be  expanded 
to  the  tri-Service  area.  Then,  DoD-wide  avoidance  in  training  dollars  through 
use  of  ASVAB  would  be  approximately  180  million  dollars  annually. 


ENLISTED  FORCE  PERSON- JOB  MATCH 


Need 


It  is  essential  that  the  Air  Force  have  effective  mathematical  algorithms 
and  operational  software  that  yield  more  nearly  optimum  person-job  matches 
(PJM) ,  i.e. ,  assign  the  right  person  to  the  right  job.  The  matching  of  person 
characteristics  and  capabilities  to  skill  reguirements  is  the  key  inaredient 
in  the  classification  of  Air  Force  personnel.  The  Air  Force  needed  to  develop 
consistent  classification  procedures  that  would  satisfy  preentry  and  postentry 
phases  of  the  enlistment  process. 

In  December  1971,  the  Air  Force  implemented  an  assignment  system  for  new 
enlistees — the  Procurement  Management  Information  System  (PROMIS).  This  sys¬ 
tem  provided  direct  telephone  contact  between  recruiters  and  the  Recruitina 
Services  Accession  Control  Center  (ACC),  which  could  enlist  applicants  up  to  6 
months  in  advance. 

The  system  had  several  disadvantages.  First,  the  time  reauireo  for 
recruiter  interaction  with  the  ACC  greatly  exceeded  the  initial  estimates. 
This  delay  resulted  in  the  telephone  lines  becoming  saturated,  and  recruiters 
encountered  a  long  delay  before  they  could  contact  the  ACC.  Because  of  the 
batch  processing  nature  of  PROMIS,  the  job  slots  opened  up  by  cancellations, 
and  the  new  monthly  guotas  were  posted  all  at  once  at  known  hours  of  the  day 
and  days  of  the  month.  This  resulted  in  an  overload  situation  on  telephone 
lines  as  large  numbers  of  recruiters  all  over  the  country  attempted  to  secure 
contact  with  ACC  and  reserve  the  most  desirable  jobs  for  their  prospects.  For 
example,  on  February  27,  1974,  July  openings  were  posted  for  the  first  time. 
(July  is  typically  a  peak  month  for  enlistments  as  high  school  seniors 
graduate.)  Air  Force  recruiters  flooded  ACC  with  phone  calls,  and  the  over¬ 
load  began  to  tie  up  trunk  lines  for  other  telephone  users  all  over  Texas  and 
then  in  other  parts  of  the  country.  The  telephone  company  quickly  disabled 
the  ACC  inward  lines  to  prevent  reduced  service  to  its  other  customers.  A 
second  disadvantage  was  that  the  assignment  process  did  not  approximate  an 
optimal  solution  in  that  only  minimum  cuttina  scores  were  reauired.  Third, 
the  system  did  not  provide  recruiters  with  an  up-to-the-minute  status  and 
management  information  (e.g.,  auotas  were  not  immediately  available  nn 
request) . 

With  the  advent  of  the  zero-draft  environment,  and  the  disadvantages  of 
the  original  FROMIS  system,  the  Air  Force  identified  the  need  for  developing  a 
computer-based  assignment  system.  Initially,  a  small  computer-based  job 
reservation  system  was  developed  by  the  Air  Force  Human  Resources  Laboratory 
(AFFRL).  This  successful  effort  demonstrated  the  feasibility  of  developing  a 
large-scale  computer-based  PJM  system  with  AFHRL  serving  as  the  primary 
developer  of  the  PJM  specifications. 

R&D  Solution 

flfhFL  completed  this  effort  in  response  to  a  request  from  Air  Force 
Recruiting  Service  and  the  Personnel  Procurement  Directorate  of  the  Air  Force 
Manpower  and  Personnel  Center.  The  general  concept  of  a  personnel  selection 
and  placement  system  is  depicted  in  the  following  diagram.  This  concept 
served  as  the  basis  for  the  formulation  of  a  personnel  job  assignment  model 
which  was  the  framework  for  the  AFHRL  preenlistment  PJM  research  and 
development . 
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The  approach  involved,  first,  the  establishment  of  a  Job  Properties  Array 
that  consisted  of  attributes  or  characteristics  associated  with  jobs.  Second, 
a  Person  Characteristics  Array  was  established,  consisting  of  those  attributes 
of  the  individual  applicants  that  wojld  be  linked  to  job  criteria.  Once  these 
two  arrays  were  available,  the  third  step  was  to  match  the  Job  Properties  with 
the  People  Characteristics.  This,  in  turn,  as  the  final  step,  gave  a 
Predicted  Payoff  Array  with  each  cell  of  the  array  representing  the  worth  or 
"payoff"  of  a  given  individual  in  a  given  job. 

Although  the  assignment  process  could  be  accomplished  from  the  Predicted 
Payoff  Array,  that  process  would  not  always  assign  a  person  to  the  job  that 
would  give  the  hicfiest  payoff  to  the  Air  Force  since  many  persons  might  be 
considered  for  the  job.  Therefore,  the  Predicted  Payoff  Array  was  converted 
to  an  Allocation  Array  which  contained  allocation  indicators  that  reflected 
the  desirability  of  each  assignment  for  overall  Air  Force  effectiveness. 

The  predicted  payoff  values  assigned  to  each  job  for  which  a  person  was 
eligible  were  generated  using  a  policy-specifying  process  and  included  an 
aptitude-job  difficulty  component,  a  technical  school  success  component,  a  iob 
area  performance  component,  a  variable  fill  component,  and  a  minority/ 
non-minority  component.  Once  the  payoff  eguations  were  developed  for  all  Air 
Force  specialties,  a  PJM  prototype  was  demonstrated. 

The  pre-enlistment  PJM  system  selects  and  assigns  recruits  to  specific  Air 
Force  specialties  or  to  one  of  the  aptitude  areas  (i.e.,  mechanical,  adminis¬ 
trative,  general,  or  electronics).  In  addition  to  the  development  of  a 
pre-enlistment  PJM  system,  AFHRL  initiated  and  completed  R&D  on  a  post¬ 
enlistment  Person-Job  Match  system.  The  system  is  compatible  with  the 
pre-enlistment  system  and  provides  an  assignment  system  for  those  recruits  in 
basic  training  who  were  assigned  by  the  pre-enlistment  PJM  system  to  an  apti¬ 
tude  area  instead  of  to  a  specific  Air  Force  specialty.  Once  in  basic  train¬ 
ing,  individuals  receive  their  specific  jobs  via  the  post-enlistment  PJM 
system.  The  post-enlistment  system  contains  the  additional  variables  of 
Preference  Match  and  Interviewer  Rating.  Preference  Match  is  the  degree  to 
which  specialty  matches  trainee's  specialty  preferences  collected  during  Basic 
Military  Training  (BMT) .  Interviewer  Rating  is  the  trainee's  rating  by  a  BMT 
classification  interviewer  on  trainee's  general  potential  for  service  success. 

Through  the  AFFRL  R&D,  progress  has  been  made  in  advancing  the  state  of 
the  art  in  PJM  systems.  Some  of  the  highlights  include  the  development  of 
transportation  algorithms  for  solving  batch  assignment  problems,  the  implemen¬ 
tation  of  new  research  findings  into  the  operational  PJM  system,  the  develop¬ 
ment  of  a  post-enlistment  payoff  generation  system  similar  in  style  to  the 
pre-enlistment  system,  and  the  advancement  of  the  state-of-the-art  in  policy 
development  technigues. 


Application 

On  1  November  1976,  the  PJM  system  became  operational  for  preenlistment 
classification  with  Air  Force  representatives  at  66  Armed  Forces  Examining  and 
Entrance  Stations  (now  called  Military  Entrance  Processing  Stations)  linked  by 
remote  terminals  to  a  Burroughs  6700  computer  located  at  Randolph  AFB ,  Texas. 
The  PROMIS,  which  now  included  the  PJM  program,  was  renamed  the  Advanced 
Personnel  Data  System  Procurement  Management  Information  System  (APDS-PROMIS) . 
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An  option  given  to  a  recruit  is  to  allow  the  naming  of  a  specific  jot 
preference,  if  any.  If  the  recruit  is  eligible,  the  job  will  be  one  of  the  15 
or  16  jobs  from  which  the  recruit  can  select. 

The  recruit  is  shown  the  top  15  jobs  out  of  100  to  150  jobs  for  which  that 
recruit  is  eligible.  The  ranking  of  the  jobs  is  based  on  the  components  used 
within  the  algorithm.  Studies  have  shown  that  over  AO  percent  of  the  recruits 
selected  from  the  top  three  jobs  and  that  over  90  percent  selected  from  the 
top  15  jobs.  Only  about  10  percent  of  the  recruits  selected  jobs  from  the 
other  100  to  150  jobs.  The  system  is  a  vast  improvement  over  the  pre-PROMIS 
manual  system  since  that  system  could  only  look  at  and  offer  a  choice  from  a 
small  number  of  available  jobs.  The  pre-PROMIS  system  was  operated  manually 
by  individuals  who  would  look  up  jobs  for  recruiters  calling  in  by  phone  from 
all  over  the  United  States.  Every  job  for  which  a  recruit  was  eligible  could 
not  be  looked  up,  ranked,  and  presented  to  the  recruit  (a  15-second  or  less 
task  on  the  present  system).  The  pre-PROMIS  system  was  set  up  to  fill  jobs 
and  did  not  attempt  to  make  optimal  nor  even  good  assignments. 

The  pre-enlistment  PJM  subsystem  currently  classifies  approximately  60 
percent  of  all  Air  Force  recruits  into  guaranteed  Air  Force  jobs  in  the 
pre-enlistment  environment.  The  improved  and  expanded  PJM  subsystem  for 
post-enlistment  classification  was  developed  in  FY80  and  is  close  to  beina 
implemented  operationally  by  the  Air  Force  Recruiting  Service.  This  new 
system  will  replace  the  current  post-enlistment  classification  system,  called 
the  Processing  and  Classification  of  Enlistees,  and  will  optimally  classify 
the  remaining  AO  percent  of  recruits  who  reauire  job  classification  during  BMT. 

Payoff 

The  Air  Force  Recruiting  Service  estimated  that  the  use  of  the  PJM  resul¬ 
ted  in  1.7  million  dollars  in  direct  cost  savings  by  eliminating  over  100 
unnecessary  positions  and  by  phasing  out  old  equipment.  A  .75  million  dollar 
cost  avoidance  resulted  from  bypassing  expenditures  budgeted  for  the  previous 
system.  The  cost  of  the  new  system  was  2  million  dollars.  Therefore,  the 
overall  cost  savings  was  .A5  million  dollars.  Although  dollar  savings  were 
minimal,  compared  to  the  overall  cost  of  the  recruiting  program,  the  most 
important  benefit  to  the  Air  Force  was  a  more  effective  work  force  and 
improved  operational  readiness.  Furthermore,  this  system  became  the  basis  for 
the  Air  Force  Guaranteed  Enlistment  Program  and  greatly  contributes  to  a 
favorable  Air  Force  recruiting  image. 

Through  this  AFI-FL  R&D,  the  Air  Force  is  now  provided  with  a  computer- 
based  assignment  capability  superior  to  any  previous  system.  The  PJM  system 
is  much  better  in  terms  of  response  time  and  provides  a  more  nearly  optima) 
assignment  process  than  had  been  previously  available.  As  an  integral  part  of 
APDS-PROMIS,  PJM  continues  to  be  the  primary  component  of  that  system.  Since 
implementation  in  1976,  the  Air  Force  Recruiting  Service  personnel  have  run 
approximately  two  million  PJM  searches. 
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APTITUDE  REQUIREMENTS  FOR  AIR  FORCE  JOBS 


Need 


One  of  the  major  organizational  goals  of  the  Air  Force,  and  one  of  the 
greatest  challenges  for  management,  is  ensuring  that  the  most  talented 
enlistees  are  assigied  to  the  most  demanding  occupations.  If  high-aptitude 
personnel  are  assigned  to  jobs  having  low-aptitude  requirements,  talent  will 
be  wasted.  On  the  other  hand,  if  low-aptitude  personnel  are  assigned  to  jobs 
having  high-aptitude  requirements,  poor  job  performance  will  result.  To  the 
extent  that  manpower  resources  are  not  effectively  utilized,  operational 
readiness  and  mission  effectiveness  will  be  adversely  impacted.  Effective 
utilization  of  manpower  becomes  especially  critical  given  che  prospect  of 
talent  shortages.  This  is  precisely  the  situation  presently  faced  by  the  Air 
Force.  Decreases  in  both  the  quantity  and  quality  of  personnel  among  primary 
recruiting  age  groups  are  projected  for  the  1980's.  Consequently,  effective 
talent  utilization  will  more  frequently  surface  as  a  pressing  operational 
need.  Continuous  R&D  of  techniques  to  ensure  effective  utilization  of  man¬ 
power  resources  must  be  accomplished  in  order  to  prepare  for  a  more  effective 
Air  Force. 

Effective  manpower  utilization  can  be  accomplished  through  the  optimal 
allocation  of  talent.  This  requires,  among  other  considerations,  the  measure¬ 
ment  of  both  enlistee  aptitudes  and  job  aptitude  requirements.  Although 
satisfactory  procedures  are  available  to  measure  personnel  aptitudes  accur¬ 
ately,  procedures  used  to  establish  job  aptitude  requirements  were  both 
unsystematic  and  subject  to  influence  by  extraneous  factors.  There  was  a 
definite  need  for  a  more  reliable,  quantitative  procedure  for  establishing  job 
aptitude  requirements.  In  response  to  this  need,  the  Air  Force  Human 
Resources  Laboratory  (AFHRL)  initiated  research  to  develop  an  empirically 
based,  job-oriented  methodology  for  establishing  occupational  aptitude 
requirements. 


R&D  Solution 


This  effort  was  conducted  in  response  to  a  request  from  the  Classification 
Branch  of  the  Air  Force  Manpower  and  Personnel  Center.  The  R&D  approach 
involved  the  derivation  of  measures  of  the  learning  difficulty  associated  with 
each  enlisted  occupation.  Occupational  learning  difficulty  was  defined  as 
"the  time  it  takes  to  learn  to  perform  required  occupational  tasks  satis¬ 
factorily."  These  measures  were  to  serve  as  a  job-centered  frame  of  reference 
for  use  in  the  process  of  establishing  aptitude  requirements.  Alignment  of 
aptitude  requirements  so  as  to  correspond  to  the  order  of  occupations  in  terms 
of  learning  difficulty  would  ensure  that  the  most  talented  enlistees  are 
assigned  to  those  Air  Force  occupations  having  the  highest  learning  loads. 
Measures  of  learning  difficulty  were  based  on  task-by-task  evaluations  by  job 
supervisors  and  expert  job  analysts  and  were  generated  on  a  position-by¬ 
position  basis.  Research  has  indicated  that  ratings  of  task  learning  dif¬ 
ficulty  are  reliable.  High  levels  of  interrater  agreement  are  demonstrated 
when  task  difficulty  is  independently  evaluated  by  job  supervisors.  The 
validity  of  the  task  rating  procedure  was  established  by  demonstrating  sub¬ 
stantial  convergence  of  ratings  when  judgments  issue  from  independent  sources 
and  different  measurement  methods. 
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To  date,  occupational  learning  difficulty  has  been  derived  for  over  200 
enlisted  specialties.  This  involved  on-site  observations  of  learning  diffi¬ 
culty  for  over  10,000  tasks  and  the  calculation  of  learning  difficulty  for 
over  100,000  tasks  for  more  than  170,000  worker  positions. 

Application 

This  R&D  on  aptitude  requirements  has  produced  a  reliable,  Quantitative 
methodology  which  has  valuable  applications  for  manpower  management.  It 
yields  comprehensive  measures  of  occupational  learning  difficulty  for  use  in 
establishing  aptitude  requirements.  The  implementation  of  the  methodology 
provides  major  advantages  over  the  previous  system.  Air  Force  managers  now 
have  available  comprehensive,  job-centered  information  representative  of  the 
occupation  as  it  occurs  in  the  field.  The  methodology  also  permits  systematic 
updates  of  aptitude  requirements  given  future  changes  in  occupations  resulting 
from  occupational  restructuring,  redesign,  and/or  the  introduction  of  new 
tools  and  equipment. 

In  February  1981,  Air  Training  Command  and  the  Air  Force  Manpower  and 
Personnel  Center  established  an  interorganization  working  group  consisting  of 
representatives  from  the  Air  Force  classification,  testing,  recruiting,  train¬ 
ing,  and  research  communities.  The  objective  of  the  working  group  was  to 
evaluate  occupational  aptitude  requirements  for  all  enlisted  Air  Force 
specialties.  An  evaluation  of  aptitude  requirements  on  this  scale  had  not 
been  accomplished  since  the  early  1950' s. 

Measures  of  learning  difficulty  for  200  enlisted  occupations  were  trans¬ 
ferred  to  the  aptitude  requirements  working  group  for  use  in  their  evalua¬ 
tion.  These  measures  were  considered  along  with  information  from  the  train¬ 
ing,  recruiting,  and  operational  communities  for  the  purpose  of  restating  the 
minimum  aptitude  requirements  published  in  the  Airman  Classification  Regula¬ 
tion,  Air  Force  Regulation  (AFR)  39-1.  Aptitude  requirements  were  adjusted 
based  on  the  policy  that  the  more  difficult  the  occupation  is  to  learn,  as 
indicated  by  occupational  learning  difficulty,  the  higher  the  associated 
aptitude  requirement  should  be. 

The  result  of  the  evaluation  was  that  aptitude  requirements  for  over  150 
Air  Force  specialties  have  been  reaffirmed  or  adjusted.  Revised  aptitude 
requirements  were  published  in  the  30  April  1982  update  of  Air  Force 
Regulation  39-1.  The  working  group  responsible  for  evaluation  of  minimum 
aptitude  requirements  stated  that  the  establishment  of  mental  requirements  on 
the  basis  of  such  an  empirical,  job-centered  frame  of  reference  had  never 
before  been  accomplished.  Aptitude  requirements  R&D  will  play  a  significant 
role  in  the  future  optimal  allocation  of  Air  Force  manpower  resources. 

Payoff 

Aptitude  requirements  are  the  center  of  the  personnel  system.  Adjustments 
ih  aptitude  requirements  have  effects  on  the  personnel  classification,  train¬ 
ing,  and  recruiting  communities.  Consequently,  the  impact  of  major  adjust¬ 
ments  in  aptitude  requirements  must  be  assessed.  Research  is  continuing  in 
this  area  with  studies  being  conducted  in  the  areas  of  recruiting  and  techni¬ 
cal  training.  Investigations  are  underway  to  assess  the  impact  of  aptitude 
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requirement  adjustments  on  initial  duty  fill  and  should  provide  information 
concerning  the  impact  of  aptitude  requirement  adjustments  on  recruiting  suffi¬ 
cient  numbers  of  qualified  personnel.  Other  research  is  devoted  to  deter¬ 
mining  the  impact  on  technical  training  of  modifications  in  the  aptitudes  of 
incoming  students  and  should  provide  information  concerning  adjustments,  in 
technical  training  courses,  that  are  necessary  to  accommodate  trainees  having 
either  hither  or  lower  aptitudes. 
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FIRST -TERM  ENLISTED  FORCE  COST  STUDY 


Need 

The  first-term  enlisted  force  composes  nearly  50  percent  of  the  manpower 
in  the  Air  Force,  and  it  is  in  the  first  term  that  the  greatest  degree  of 
managerial  flexibility  for  structuring  the  enlisted  force  is  provided.  Each 
year,  operational  decisions  must  be  made  concerning  the  numbers  and  kinds  of 
people  that  should  be  entered  into  the  force  on  one  hand  and  that  should  be 
reenlisted  into  the  career  force  on  the  other  hand.  Recurring  Congressional 
and  DoD  concern  for  the  All-Volunteer  Force  has  focused  additional  attention 
on  first-term  enlistees.  Annually,  during  the  budget  planning  process,  Ques¬ 
tions  are  raised  focusing  on  both  the  management  and  cost  effectiveness  of 
various  aspects  of  the  first-term  force  such  as  attrition,  enlistment 
standards,  and  Quality  of  accessions.  These  issues  raise  extremely  difficult 
economic  Questions. 

The  Air  Force  has  many  models  that  address  various  aspects  of  the  enlisted 
force.  However,  none  of  them  provides  both  a  Quick  response  gaming  capability 
and  sufficient  detail  to  show  sensitivity  to  changing  various  first -term  para¬ 
meters.  All  the  models  are  limited  by  their  exclusion  of  a  detailed  cost 
module  for  handling  the  various  costs  associated  with  the  first-term  enlist¬ 
ment  period. 

To  resolve  these  problems,  the  Air  Force  Human  Resources  Laboratory 
(AFFRL)  initiated  research  and  development  (R&D)  to  identify  and  develop  a 
compilation  of  costs  and  attrition  data  to  assist  Air  Staff  in  addressina 
economic  and  operational  Questions,  and  in  analyzing  the  variable  interrela¬ 
tions  pertinent  to  the  first-term  enlisted  force. 

R&D  Solution 


This  effort  was  conducted  by  the  Manpower  and  Personnel  Division  of  AFHRl 
in  response  to  a  request  from  HO  USAF,  Directorate  of  Personnel  Plans, 
Analysis  Division.  This  R&D  project  consisted  of  two  pnases.  Phase  I  was 
performed  in  coordination  with  the  requesting  agency  and  was  directed  at 
establishing  the  scope  of  the  stuay  by  identifying  the  specific  cost  factors 
and  economic  parameters  commonly  used  in  policy  and  cost  analyses.  A  preli¬ 
minary  categorization  was  developed  and  reviewed  by  HQ  USAF  to  ascertain  what 
data  were  available  from  the  Cost  and  Management  Analysis  Office  of  Air  Train¬ 
ing  Command.  In  Phase  II,  a  collection  and  analysis  of  existing  economic  and 
cost  data  and  data-collection  systems  relating  to  the  first -term  enlisted 
force  were  made. 

The  product  of  this  effort  is  a  handbook  which  assembles  available  Air 
Training  Command  cost  and  attrition  data  for  each  entry-level  Air  Force 
Specialty.  Costs  related  to  recruiting,  basic  military  training,  and  initial 
technical  school  training  are  included,  as  well  as  travel  costs  to  first  duty 
station.  When  possible,  costs  are  also  Identified  as  marginal,  average, 
fixed,  total,  variable,  direct,  indirect,  or  a  combination  of  these.  An 
examination  of  the  rationale  for  computation  of  several  of  the  costs  was  also 
made.  A  proposed  computational  scheme  for  determining  the  cost  of  successful 
and  nonsuccessful  enlistee  training  was  also  presented. 
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The  results  of  this  effort  have  provided  immediate  benefit  to  decision 
makers  within  Headquarters,  Air  Force.  The  handbook  of  first-term  economic 
and  cost  data  was  delivered  to  Headquarters,  Air  Force,  in  September  1981. 
The  cost  handbook  assembles  under  one  cover  a  wealth  of  diverse  and  difficult- 
to- retrieve  cost  data  displayed  in  a  consistent,  easily  understood  format. 

Cost  factors  presented  in  the  handbook  were  used  as  the  basis  for  a  cost 
study  of  optimal  terms  of  enlistment  for  each  Air  Force  specialty.  As  a 
result  of  this  study,  6-year  enlistments  were  opened  up  to  more  Air  Force 
Specialty  Codes,  since  it  was  more  cost  effective  to  retain  airmen  than  to 
train  new  recruits  in  these  specialties.  The  cost-factor  data  have  also  been 
utilized,  specifically,  as  input  to  an  enlisted  force  model  being  designed  and 
programmed  by  the  Analysis  Division  of  the  Directorate  of  Personnel  Plans, 
Headquarters,  Air  Force.  This  model  relates  manhour  production  to  pay  tables 
to  determine  the  cost  of  replacing  recruits. 

Payoff 


By  providing  a  more  accurate  and  dependable  collection  of  costs  relative 
to  the  accession  and  training  of  recruits,  the  AFhRL  R&D  has  enhanced  the  Air 
Force  capability  to  respond  quickly  to  high-level  policy  and  budget  issues  and 
has  provided  researchers  and  analysts  with  a  tool  by  which  they  could  assess 
the  economic  implications  of  various  policy  options  in  the  management  of  the 
first-term  enlisted  force. 


retraining  and  transferability  of  skills 


Need 


Air  Force  managers  rely  heavily  on  the  capability  to  retrain  enlistees 
from  one  occupational  specialty  to  another  in  coping  with  the  continuinq 
problem  of  manning  shortages  and  overages  in  career  fields.  The  Air  Force 
retraining  program  is  an  essential  part  of  the  airman  personnel  management 
system. 

Approximately  10,000  airmen  retrain  annually,  and  Air  Force  manaoers 
expect  this  number  to  continue  during  the  foreseeable  future  as  a  result  of 
improvements  in  equipment  and  weapon  systems  and  the  attendant  changes  in  per¬ 
sonnel  requirements.  Retrainees  fall  into  a  variety  of  subgroups,  includina 
personnel  who  disqualify  in  their  current  skill,  personnel  who  retrain  in  con¬ 
junction  with  reenlistment,  other  voluntary  retrainees  from  overage  career 
fields,  personnel  selected  for  involuntary  training,  personnel  in  the  Con¬ 
tinental  united  States  (CONUS) /Overseas  imbalance  specialties  who  are  surplus 
to  CONUS  requirements,  and  technical  school  eliminees.  Retraining  occurs  at 
virtually  every  years-of-service  point. 

Once  a  retrainee  qualifies  at  the  appropriate  skill  level  in  the  new 
military  specialty,  there  is  no  identification  of  the  individual  as  a  re- 
trainee  in  the  personnel  management  system.  Consequently,  except  for  recent 
retrainees,  it  has  been  impossible  to  track  retrainees  as  a  group  and  assess 
their  performance  in  their  new  assignments.  Despite  the  substantial  contri¬ 
bution  of  the  retraining  program  to  personnel  management,  the  performance, 
progress,  and  accommodation  of  retrainees  to  their  new  occupations  have  not 
previously  been  systematically  studied.  Research  and  development  (R&D)  was 
needed  to  overcome  this  deficiency  and  to  provide  a  base  of  experience  from 
which  to  proceed.  The  opportunity  to  evaluate  the  eff;ciency  of  the  opera¬ 
tional  retraining  system  was  needed  to  determine  the  types  of  retrainino 

actions  that  are  operating  smoothly  and  the  actions  that  are  generatino 

adjustment  and  performance  problems. 

R&D  Solution 


This  effort  was  carried  out  at  the  request  of  the  Directorate  of  Personnel 
Programs,  Headquarters,  Air  Force.  A  comprehensive  evaluation  program  that 

systematically  tracked  the  performance  of  enlisted  retrainees  and  their 
progress  in  their  new  occupations  was  initiated.  The  structure  of  the  study 
was  organized  around  four  principal  issues  identified  as  having  to  do  with  the 
retrainee's  progress  in  the  new  specialty  beginning  with  technical  training, 
subsequent  on-the-job  performance  and  adjustment,  and  overall  career  progres¬ 
sion.  Of  special  interest  was  the  distinction  between  voluntary  and  selective 
retraining  actions. 

Information  on  retrainees  was  obtained  from  two  sources.  Historical  per¬ 
sonnel  data  from  1973  to  1979  maintained  at.  the  Air  Force  Human  Resources 
Laboratory  served  as  the  basis  for  evaluating  technical  school  outcomes  and 

skill  and  grade  progression.  In  addition,  a  large  scale  field  survey  was 

conducted  in  1980.  Retrainees  and  their  supervisors  responded  to  inquiries 

about  performance  and  adjustment  and  selective  retraining  issues. 
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Ihe  historical  data  permitted  comparisons  of  all  tp trainees  who  atteiuied 
Dasic  technical  schools  between  1973  to  197/  and  nonpriur-service  airmen 
attending  the  same  schools  during  the  same  time.  The  samples  included  19,881; 
retrainees  and  231,317  nonretrainees.  Two  criteria  were  defined:  final 
school  grade  and  attrition.  Rather  than  just  a  gross  comparison  of  the  two 
groups,  investigators  took  into  account  the  effects  of  aptitude  scores  that 
have  been  shown  to  influence  performance  in  technical  school.  The  aptitude 
variable  was  also  important  because  of  the  present  policy  to  permit  waivers  of 
minimum  aptitude  scores  for  retrainees.  For  the  retrainee  group,  additional 
information  was  collected  about  their  time  in  service  and  background  experi¬ 
ence  at  retraining.  Researchers  expected  that  some  optimum  time  in  service  or 
particular  type  of  background  experience  would  increase  the  chances  of  success 
in  technical  school. 

Results  of  this  study  provided  empirical  support  for  the  viability  of  the 
retraining  program.  In  technical  school,  at  each  aptitude  level,  retrainees 
typically  performed  better  than  nonretrainees  both  in  terms  of  percent  passina 
the  course  and  in  the  achievement  levels  of  those  who  did  pass.  Retrainees 

achieved  course  grade  levels  that  justify  the  current  10  point  discount  or 

aptitude  waiver. 

The  other  major  findings  with  regard  to  technical  school  performance  had 
implications  for  who  should  be  retrained  and  when.  As  would  be  expected, 
retrainees  with  higher  aptitudes  performed  better.  The  amount  of  time  in 
service  before  retraining  also  influenced  performance,  with  more  service  time 
leading  to  better  expectations  for  success  up  to  a  point  at  about  the  end  of 

the  retrainee's  fourth  tour.  Type  of  background  experience  was  also  shown  to 

have  an  effect.  Airmen  who  transferred  between  Air  Force  specialties  in  the 
same  aptitude  areas  had  some  advantage  over  those  who  changed  into  different 
aptitude  areas. 

The  skills,  abilities,  and  performance  of  retrainees  once  they  were  on  the 
job  in  the  new  specialty  were  also  examined.  Supervisors  contacted  through 
the  field  survey  were  asked  to  provide  job  performance  assessments  of  indivi¬ 
dual  retrained  airmen  under  their  immediate  supervision  as  well  as  of  non- 
retrained  airmen  for  comparative  analysis.  Ratings  from  supervisors  on  about 
13,000  retrainees  and  5,000  nonretrainees  were  collected.  On  these  supervisor 
ratings,  considerable  overlap  in  ratings  of  the  retrainees  and  nonretrainees 
in  terms  of  both  skills  and  abilities  and  performance  was  found.  Also,  super¬ 
visors  were  asked  to  identify  a  cut-off  point  in  time  beyond  which  non¬ 
voluntary  retraining  actions  should  not  be  considered.  Supervisors  (75 
percent)  identified  15  years  as  the  cutoff  point  for  involuntary  retrainina 
actions. 

The  next  major  issues  addressed  in  the  study  were  job  and  career  progres¬ 
sion,  which  were  evaluated  by  comparing  skill  upgrading,  promotion,  and  reten¬ 
tion.  To  accomplish  this,  an  historical  data  base  containing  about  1/2 
million  cases  for  years  1977  to  1979  was  developed.  Survey  data  provided 
supplemental  information  on  the  job  and  career  progression  issues. 

Data  revealed  that  the  time  reguired  for  retrainees  to  upgrade  was  less 
than  or  equivalent  to  that  for  nonretrainees  at  fixed  time-in-specialty 
intervals.  Survey  data  were  consistent  with  the  historical  data.  Supervisors 
reported  that  retrainees  required  both  less  time  and  less  help  to  upgrade. 
The  retrainees  themselves  felt  that  their  skill  level  was  commensurate  with 
their  technical  ability  and  job  knowledge.  Promotion  data  indicated  that, 
overall,  lower  percentages  of  re trainees  were  promoted. 


To  supplement  historical  retention  data,  information  on  reenlistment  was 
collected  from  the  field  survey.  When  asked  what  impact  retraining  had  on 
likelihood  of  reenlistment,  both  supervisors  and  retrainees  responded  that 
retraining  had  a  positive  effect.  That  is,  the  likelihood  of  reenlistment 
typically  increased  after  retraining. 

Another  issue  was  retrainees '  adjustment  to  their  new  specialty.  Briefly, 
supervisors  gave  both  retrainees  and  nonretrainees  equally  high  marks  on  the 
adjustment  indicators. 

A  final  issue  was  selective  retraining  with  particular  emphasis  on  a 
comparative  assessment  of  voluntary  versus  selective  retraining  actions.  The 
findings  indicated  that  selective  retraining  impacts  negatively  on  morale, 
motivation,  and  productivity,  but  that  there  were  management  factors  which 
could  be  employed  to  attenuate  the  negative  impact. 

Application 

The  Directorate  of  Personnel  Programs  has  initiated  policy  and  procedural 
changes  based  on  the  results  of  this  research.  It  has  brought  about  a 
refocusing  of  the  Air  Force  Retraining  Advisory.  In  the  past,  the  Advisory 
consisted  essentially  of  a  compilation  of  shortages,  by  grade,  in  the  various 
specialties.  Using  this  approach,  typically  20,000  to  30,000  shortages  were 
listed,  whereas  only  around  10,000  retraining  actions  per  year  occurred. 
Since  most  retraining  actions  are  voluntary,  this  approach  did  little  to 
ensure  that  retrainees  flowed  into  the  skills  where  the  greatest  need 
existed.  The  Advisory  is  now  more  carefully  focused,  limiting  the  shortage 
list  to  the  anticipated  total  size  of  the  retraining  group,  placing  the 
retraining  slots  in  the  skills  with  the  most  severe  grade  shortages. 

A  recent  change  to  the  Retraining  Program  was  supported  by  data  from  this 
R&D  effort.  Previously,  nonvolunteers  were  retrained  regardless  of  time  in 
service.  Retraining  decisions  now  include  the  addition  of  a  time  in  service 
cutoff  (13  years)  for  retraining  nonvolunteers.  This  decision  was  based  on 
the  75-percent  supervisor  agreement  on  15  years  as  a  maximum  cutoff  for  non¬ 
voluntary  retraining. 

Payoff 

Improvements  in  the  selection/approval  of  retrainees  based  on  this  R&D 
effort  will  increase  technical  school  graduation  rates  and  will  result  in 
higher  morale  and  better  job  performance.  Assignments  that  optimize  skills 
transfer  will  result  in  dollar  savings  through  lowered  attrition  as  well  as 
through  reductions  in  training  time  reauired  for  retrained  personnel  to 
achieve  proficiency  in  their  new  occupations. 


AIR  FORCE  READING  ABILITIES  TEST 


Need 


Illiteracy  problems  in  the  military  Services  have  been  a  concern  of  the 
Department  of  Defense  for  years.  Many  Air  Force  organizations  have  beer 
administering  commercially  published  reading  tests  to  military  personnel.  The 
Army  and  the  Navy  also  have  relied  on  commercial  reading  tests.  These  tests 
have  been  used  for  assignment  to  remedial  training  programs,  for  counseling 
students,  and  for  determining  reading  levels  of  airmen  in  various  occupational 
specialties.  However,  because  as  many  as  12  different  reading  tests  were  used 
by  Air  Force  agencies,  it  was  impossible  to  make  comparisons  across  groups  or 
to  provide  uniformity  of  treatment  for  personnel. 

Consideration  was  given  to  adopting  a  single  commercial  reading  test  for 
entire  Air  Force  use.  But  this  was  infeasible  because  Air  Force  norms  would 
still  need  to  be  established  since  the  Air  Force  is  a  more  restricted  popula¬ 
tion  than  the  national  population.  If  Air  Force  norming  was  necessary,  it  was 
a  small  further  step  to  develop  a  distinct  Air  Force  Reading  Abilities  Test 
(AFRAT).  Furthermore,  in  developing  a  unigue  reading  test,  Air  Force  managers 
would  offset  royalties  charges  to  commercial  publishers  and  would  maintain 
control  of  the  reading  test.  Thus  changes,  refinements,  and  improvements 
would  be  possible,  which  could  not  be  accomplished  with  commercial  tests. 
Also,  norms  could  be  adapted  as  conditions  changed. 

The  Air  Force  Human  Resources  Laboratory  (AFl-RL)  conducted  a  study  tc 
evaluate  the  capability  of  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  to  determine  the  reading  ability  skills  of  applicants  for  enlistment. 
The  ASVAB  and  the  Gates-MacGinitie  reading  tests  were  administered  to  2,A?2 
subjects.  A  high  correlation  of  .79  was  found  between  the  ASVAB  General 
Technical  composite  and  reading  grade  level.  It  was  concluded  that  the  ASVAB 
was  a  valid  reading  assessment  device  and  was  screening  out  most  applicant? 
with  marginal  literacy  skills. 

As  a  front-end  screening  device,  the  ASVAB  was  useful;  however,  several 
factors  made  the  ASVAB  unsatisfactory  as  a  general  purpose  reading  assessment 
instrument.  The  ASVAB  composites  contain  several  short  subtests  covering 
different  ability  factors.  The  General  composite  includes  Arithmetic  Reason¬ 
ing  in  addition  to  the  verbal  subtests  of  Word  Knowledge  and  Paragraph 
Comprehension.  Most  women  perform  slightly  better  than  do  men  on  verbal 
tests;  however,  they  generally  do  somewhat  less  well  than  do  men  on  Arithmetic 
Reasoning.  When  the  General  composite  is  used  to  gauge  reading  ability  of 
women,  underestimation  will  result  in  the  majority  of  cases. 

A  brief  reading  assessment  device  was  freauently  needed  after  enlistment. 
For  many  purposes,  administering  the  entire  ASVAB  was  too  time-consuming. 
Furthermore,  frequent  and  repeated  use  of  the  ASVAB  would  compromise  its 
primary  purpose.  For  individual  measurement,  therefore,  a  more  content 
specific  and  reliable  measure  of  reading  than  that  based  on  ASVAB  was 
desired.  A  decision  was  made  to  develop  an  Air  Force  specific  reading  abili¬ 
ties  test  in  two  equivalent  (alternate)  forms.  This  was  to  be  a  brief  reading 
assessment  device  that  could  be  administered  to  large  numbers  of  individuals. 


RAD  Solution 


This  project  w^s  conducted  by  the  Manpower  and  Personnel  Division  of  AFHRL 
in  response  to  a  reauest  from  the  Personnel  Testing  Branch  of  the  Air  Force 
Manpower  and  Personnel  Center.  The  development  and  norming  of  the  AFRAT  was 
designed  to  standardize  the  assessment  of  reading  ability  of  Air  Force  person¬ 
nel  and  replace  the  commercial  reading  tests  that  have  been  jsed  throughout 
the  Air  Force.  The  goal  of  this  effort  was  to  develop  a  readina  test  with 
appropriate  norms. 

Two  parallel  or  comparable  forms  of  AFRAT  werp  developed.  This  readinr; 
test  consists  of  45  vocabulary  items  in  a  synonym  format  and  40  comprehension 
items  consisting  of  one  or  several  paragraphs  followed  by  one  or  more  Ques¬ 
tions.  The  comprehension  items  reguire  either  paraphrasing  or  making  infer¬ 
ences  from  the  passages.  All  items  are  multiple-choice  with  four  alterna¬ 
tives,  with  a  total  test  time  limit  of  50  minutes. 

Samples  of  enlistees  totaling  12,f'83  were  tested  for  various  analyses. 
The  proportion  of  correct  item  responses  was  .85  for  both  Forms  A  and  B.  When 
correct  responses  on  individual  items  were  correlated  with  correct  responses 
on  the  total  test,  the  average  item  to  test  total  correlations  were  .60  anc 
.62  for  Forms  A  and  B,  respectively.  The  relationships  of  AFRAT  forms  to 
three  commercial  reading  tests  were  moderate-to-high  (.60  to  .67).  The  inter¬ 
relationship  between  Forms  A  and  B  was  somewhat  higher  (.73).  In  addition, 
subtest  and  total  score  variances  for  Forms  A  and  B  were  equal. 

Percentiles  were  computed  for  AFRAT  scores  and  Reading  Grade  Level  (RGL) 
scores  from  a  composite  of  three  commercial  tests.  AFRAT  forms  were  eauatec 
to  an  average  RGL  through  use  of  an  Air  Force  selection  composite  as  an  anchor 
test.  Raw-score-to-RGL  conversion  tables  for  the  4th  through  the  12th  RGL 
were  generated  for  AFRAT  subtest  and  total  scores.  In  addition,  percentile 
norms  for  Air  Force  personnel  were  generated.  Because  they  were  based  on  Air 
Force  subjects,  these  computed  percentile  and  RGL  norms  should  be  appropriate 
for  enlistees. 

A  number  of  studies  were  conducted  correlating  ’  echnical  school  arades 
with  AFRAT  scores.  The  median,  or  average,  validity  coefficient  with  techni¬ 
cal  training  grades  was  .40  for  an  experimental  version  of  AFRAT.  This 
indicates  a  moderate  but  significant  relationship.  Those  airmen  with  RGL 
scores  of  9.0  or  less  nad  an  average  training  grade  of  five  points  below  that 
of  all  other  airmen.  This  indicates  that  the  AFRAT  was  a  valid  predictor  of 
technical  training  performance  to  a  moderate  degree.  Operational  results  for 
AFRAT  will  be  monitored  to  ensure  that  the  test  does  function  as  anticipated. 

Application 

Based  on  the  norming  and  validation  studies,  the  AFRAT  was  implemented  Air 
Force  wide  in  April  1982  as  an  operational  test  supporting  many  formal  train¬ 
ing  programs  and  on-the-job  training.  It  is  used  in  Basic  Military  Training 
to  screen/identify  poor  readers  for  discharge  or  remediation.  Airmen  with 
reading  problems  can  be  identified  more  accurately,  thus  reducing  the  costs 
associated  with  training  failures  caused  by  reading  problems. 


The  AFRAT  filled  a  void.  Previously,  more  than  12  different  readinn  test' 
were  used  in  the  Air  Force,  resulting  in  a  problem  concerning  interpretation 
of  test  scores.  Now,  two  forms  of  a  single  test  are  used  to  assess  literacy 
skills.  Use  of  AFRAT  provides  a  better  assessment  of  Air  Force  student  per¬ 
sonnel  reading  ability  and  allows  for  comparison  of  groups  across  occupations 
and  major  commands,  and  individuals  over  time. 

Reading  capabilities  of  samples  of  first-term  airmen  can  be  evaluated  at 
several  points  in  order  to  develop  an  accurate  picture  of  the  nature  and 
extent  of  reading  abilities  of  Air  Force  student  personnel.  Also,  reaoinc 
abilities  of  students  who  have  taken  remedial  training  in  Basic  Military 
Training  can  be  evaluated  at  the  2-year  and  4-year  marks  to  determine  whether 
their  improved  abilities  remain  stable  or  decay  over  time.  By  using  a  sinale 
test,  comparisons  between  different  types  of  remedial  programs  can  be  made. 
Thus,  AFRAT  results  are  providing  an  important  data  base  for  future  readinc 
research. 
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Need 


As  a  result  of  the  increased  availability  and  improved  capabilities  of 
electronic  computers,  there  has  been  a  growina  interest  in  new  methods  nf 
testing.  A  joint-Service  project  is  underway  to  design,  develop,  test,  and 
evaluate  a  system  for  automated,  adaptive  administration  of  military  personnel 
selection  and  classification  tests.  The  objectives  of  this  effort  are  to 
develop  a  cost-effective  Computerized  Adaptive  Testing  (CAT)  system  suitable 
for  nation-wide  implementation  in  the  Military  Enlistment  Processing  Stations, 
and  to  design  the  system  so  that  it  can  replace  the  paper-and-pencil  Armen 
Services  Vocational  Aptitude  Battery  ( ASVAB) .  Adaptive  testing  is  the  dynamic 
tailoring  of  test  difficulty  to  each  examinee's  ability  contingent  on  perform¬ 
ance  at  earlier  stages  of  the  test.  Such  computer-generated  adaptive  tests 
should  allow  determination  of  aptitudes  with  fewer  test  items  than  with 
conventional  paper-and-pencil  tests,  thus  proving  to  be  not  only  more  precise, 
but  more  cost-effective  as  well.  A  research  and  development  (R&D)  project  was 
undertaken  to  develop  a  computerized  measurement  system  for  selection  and 
classification  uses  and  to  estimate  its  impact  on  the  effectiveness  with  which 
the  Air  Force  selects  its  new  accessions  and  places  them  on  the  job. 

R&P  Solution 


In  producing  the  CAT  system,  two  separate  elements  are  needed:  psycho¬ 
metric  construction  of  the  adaptive-testing  procedures  and  tests,  and  en¬ 
gineering  development  of  the  system  to  implement  those  procedures  and  tests. 
The  Navy  Personnel  Research  and  Development  Center  is  responsible  for  the 
engineering  development  of  the  system  and  the  Air  Force  Human  Resources 
Laboratory  (AFHTL),  for  constructing  the  tests.  The  AFHRL  effort  was  carried 
out  in  response  to  a  request  from  the  Personnel  Testing  Branch  of  the  Air 
Force  Manpower  and  Personnel  Center. 

The  psychometric  procedures  reauired  writing  computer  programs  and  sub¬ 
routines  to  accomplish  adaptive  item  presentation  and  item  scoring.  From 
this,  a  model  CAT  was  developed  and  installed  at  the  San  Antonio  Armed  Forces 
Examining  and  Entrance  Station  where  approximately  150  applicants  for  enlist¬ 
ment  were  tested  with  three  adaptive  tests:  Word  Knowledge,  Arithmetic 
Reasoning  and  Space  Perception.  Adaptive  test  scores  were  found  to  correlate 
highly  (r  -  .82)  with  ordinary  paper-and-pencil  test  scores. 

As  a  prelude  to  developing  operational  item  banks,  the  following  auestions 
posed  themselves:  (a)  How  can  the  special  item  statistics  be  estimated? 
(b)  How  can  item  banking  be  automated?  (c)  How  many  items  are  reauired  for 
an  operational  item  bank?  (d)  How  can  items  be  linked  from  separate  testings 
to  a  common  item  metric? 

The  first  of  these  auestions  asked  how  to  estimate  item  parameters  and  was 
answered  via  a  large  scale  simulation  study  comparing  four  extant  methods  on 
three  specific  ability  distributions.  Results  indicated  the  desirability  of 
rectilinear  or  normally  distributed  samples  of  about  2,000  subjects  in  order 
to  get  useful  correlations  between  true  and  estimated  item  parameters  (rac 
.89,  rbb=  -98,  rcc=  .56).  The  auestion  pertaining  to  item  banking  wac 
answered  by  developing  a  comprehensive  array  of  software  and  estimation 
procedures,  including  a  test  construction  module  and  an  item  editor  module. 


21 


An  empirical  investigation  of  the  techniques  showed  that  the  performance  of 
item  pools  could  be  accurately  estimated  for  score  means  (10.5  vs.  10.16), 
standard  deviation  (4.3  vs.  4.0)  and  reliability  (.80  vs.  .72).  A  simulation 
study  was  conducted  to  answer  the  question  of  how  many  items  are  necessary  for 
an  item  bank.  Results  showed  that  between  200  and  300  items  were  optimal, 
yielding  a  correlation  between  true  and  estimated  ability  of  .982  with  -.98 
bias  and  .294  average  absolute  difference  between  them. 

Similarly,  simulation  studies,  involving  sample  sizes  from  250  to  2,000, 
investigated  several  designs  for  linking  item  parameters.  The  results  indica¬ 
ted  specific  designs  were  advantageous  for  specific  needs  and  conditions.  It 
was  concluded  that  efficiency  losses  due  to  linking  would  be  limited  to  1  per¬ 
cent.  Based  on  this  knowledge,  operational  item  banks  in  the  nine  non-speeded 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  ability  areas  is  proceed¬ 
ing.  The  project  is  based  on  the  responses  of  over  100,000  applicants  for 
military  enlistment  and  will  be  completed  in  1983. 

Two  additional  projects  are  noteworthy.  First,  a  validity  study  of  two 
types  of  adaptive  testing  for  jet  engine  mechanics  was  conducted.  In  this 
study,  450  technical  training  students  were  given  adaptive  and  ordinary 
tests.  No  differences  in  validity  were  found,  although  the  adaptive  test 
reached  its  maximum  validity  in  1/2  to  2/3  the  number  of  items.  Finally,  an 
investigation  of  noncomputerized  adaptive  tests  was  run  which  administered  one 
of  three  prototype  paper-and-pencil  adaptive  tests  to  711  basic  airmen.  The 
areas  tested  were  Word  Knowledge  and  Arithmetic  Reasoning.  Correlations  of 
the  adaptive  test  with  the  Armed  Forces  Qualification  Test  were  .56,  .51,  and 
.63  for  the  prototype  and  equal  to  the  AFQT  paper-and-pencil  subtest  correla¬ 
tion.  The  effort  provided  a  successful  demonstration  that  adaptive  testing 
can  be  conducted  without  a  computer. 

Application 

The  AFHRL  is  a  recognized  leader  in  the  field  of  CAT.  A  goal-oriented 
series  of  efforts  is  underway  to  develop  both  prototype  and  operational  item 

pools,  as  well  as  to  advance  the  state  of  knowledge  regarding  the  theoretical 

basis  of  adaptive  testing.  These  efforts  are  directed  at  completing  the 

knowledge  as  it  applies  to  the  Laboratory's  responsibility  to  the  Joint 
Services  Computer  Adaptive  Testing  Interservice  Coordinating  Committee.  This 
R&D  is  designed  to  achieve  the  objectives  necessary  under  the  tasking  of  this 
commitment. 

CAT  is  scheduled  to  become  operational  in  FV85  and  will  eventually  permit 
the  Air  Force  and  the  other  Services  to  implement  the  ASVAB  in  an  adaptive 
testing  system.  These  efforts  support  the  DoD  scheduled  implementation  date 
and  already  provide  a  great  many  of  the  analytic  tools  needed  to  support  such 
a  system  in  the  development  of  conventional  bias  free  tests.  These  R&D 

efforts  are  viewed  as  covering  the  first  steps  in  the  development  of  a 
computer-based  selection  and  classification  system  using  adaptive  testina 
concepts.  Successful  implementation  of  CAT  for  ASVAB  usage  will  be  followed 
bv  application  of  perceptual  motor  ability  tests,  as  well  as  the  traditional 
paper-and-pencil  aptitude-type  tests.  Interest  and  attitude  measurement  could 
also  more  easily  be  incorporated  than  is  possible  with  paper-and-pencil  tests. 
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More  than  4  hojrs  are  required  to  prepare  and  administer  the  ASVAP  fen'"  ; 
testing  alone  takes  about  2-3/4  hours.  Tailoring  the  test  to  the  individual 
would  reduce  current  test  duration  by  50  percent  or  more.  Replacement  of  all 
printed  test  material  by  electronic  media  would  eliminate  the  susceptibility 
to  errors  inherent  in  manual  score  handling.  Implementation  of  an  efficient 
and  valid  computerized  measurement  system  would  add  precision  to  the  decision 
made  concerning  the  selection  and  job  placement  of  new  accessions  and  promises 
to  be  more  cost-effective  than  conventional  paper-and-pencil  tests,  especially 
by  reducing  testing  time  and  the  possibility  of  misclassification,  or  need  for 
later  attrition  or  reclassification  and  retraining.  Logically,  CAT  should 
reduce  test  compromise  and  provide  greater  sensitivity  across  the  full  range 
of  examinees'  aptitudes.  The  computerized  measurement  system  could  be  admin¬ 
istered  during  enlistment  processing  by  all  the  Services.  These  projected 
benefits  will  be  tested  for  empirical  validation  in  further  R&D  efforts. 


ORGANIZATIONAL  ASSESSMENT  PACKACF 


Need 


Within  organizations,  management  personnel  are  concerned  with  how  well 
their  organization  meets  its  objectives.  These  objectives  are  freauently 
measured  in  terms  of  productivity,  cost  savings,  and  retention  of  personnel. 
The  Air  Force  is  vitally  concerned  with  the  organizational  effectiveness  of 
its  units.  The  Leadership  and  Management  Development  Center  (LMOC)  has  thp 
mission  of  providing  consultative  services  to  Air  Force  installations  through¬ 
out  the  world  in  order  to  increase  organizational  effect iveness.  In  addition, 
LMDC  is  responsible  for  providing  leadership  and  management  education  Air 
Force-wide.  A  methodology  was  needed  for  measuring  organization  effectiveness 
and  identifying  leadership  and  manaaement  problem  areas.  The  primary  use  of 
the  instrument  would  be  to  diagnose  organizational  climate  and  indicate 
possible  problem  areas  for  further  attention.  The  results  of  the  evaluative 
technigues  would  also  provide  curriculum  input  in  terms  of  the  types  of 
leader ship/manaqement  skills  that  need  to  be  taught  or  enhanced  and  the 
leadership  styles  that  are  effective  in  different  Air  Force  situational 
environments  (i.e.,  different  types  of  organizations).  An  assessment  packaoe 
was  needed  to  provide  means  of  gathering  leadership/management  style,  situa¬ 
tional  environment,  and  criteria  data  and  of  relatina  each  to  the  other. 

R&D  Solution 


This  project  was  conducted  by  the  Manpower  and  Personnel  Division  of  the 
Air  Force  Human  Resources  Laboratory  (AFtfTL)  in  response  to  a  reauest  from  the 
Leadership  and  Management  Development  Center  of  the  Air  Training  Command.  A 
1976  study  by  Hendrix  developed  the  Three  Component  Leadership  Effectiveness 
Model.  This  model  considered  organizational  effectiveness  to  be  a  function  of 
the  criterion  selected,  the  managerial  style  employed,  and  the  situational 
environment — which  includes  the  manager's  subordinates,  peers,  and  other 
personnel  in  the  environments.  This  contingency  model  was  tested  by  a  com¬ 
puter  simulation.  Situations  were  created  by  presenting  a  leader  with  a  six- 
factor  profile  that  depicted  the  situation.  Next  five  proposed  leadership 
styles  were  presented  from  which  the  leader  selected  one  as  being  most 
appropriate  for  the  situation  and  implemented  it  (i.e.,  input  the  selection 
into  a  computer).  Feedback  to  the  operator  (by  computer  simulation  output) 
then  told  how  successful  the  leader's  performance  was.  After  exposure  to  ?0C 
situational  profiles,  the  leader's  behavior  was  captured  using  a  regression 
model.  An  individual's  policy,  once  captured,  could  then  be  applied  to  new 
situations  to  see  if  it  was  a  valid  predictor  model. 

The  Organizational  Assessment  Package  (OAP)  was  desianed  to  measure  the 
basic  components  of  the  Three  Component  Organizational  Effectiveness  Model. 
The  Supervisory  Job  Inventory  was  designed  to  measure  manaaerial  style,  while 
the  situational  environment  is  measured  by  two  sections  of  the  OAP:  the 
Background  Information  section  and  the  Organizational  Job  Inventory.  The 
criteria  are  satisfaction,  organizational  climate,  and  perceived  produc¬ 
tivity.  These  are  measured  by  the  sections  entitled  Job  Satisfaction 
Questionnaire,  Organizational  Climate  Inventory,  and  Perceived  Productivity 
Index. 


Items  within  each  of  the  OAP  sections  have  been  written  to  measure  certain 
factors.  The  Background  Information  section  contains  biographical  information 
items  and  items  associated  with  factors  in  the  situational  environment.  The 
factors  in  the  situational  environment  that  the  items  attempted  to  measure 
included  (a)  organizational  level  of  work  group,  (b)  work  group  type,  (c)  work 
group  size,  (d)  group  member  maturity,  (e)  organization's  geographic  region, 
(f)  extent  to  which  work  group  meetings  are  used  to  establish  goals,  (a) 
extent  of  communication  between  work  group  members,  and  (h)  stability  of  work 
hours. 

The  situational  environment  was  in  part  measured  by  the  Job  Inventory. 
The  Job  Inventory  centers  on  five  basic  factors  (Core  Job  Dimensions):  (a) 
skill  variety,  (b)  task  identity,  (c)  task  significance,  (d)  autonomy,  and  (e) 
feedback  from  the  job.  These  factors  were  measured  by  the  Job  Inventory,  plus 
one  additional  work- related  factor,  (labeled  Work  Interference).  This  addi¬ 
tional  factor  deals  with  the  extent  and  adeguacy  of  (a)  added  duties,  (b) 
eouipment  and  supplies,  and  (c)  provided  work  space. 

In  the  criterion  area,  organizational  climate  was  measured  by  the 
Organizational  Climate  Inventory,  which  included  the  following  hypothesized 
factors:  (a)  communications,  (b)  general  organizational  conditions,  (c) 

employee  concern,  (d)  employee  commitment,  (e)  decision  making,  and  (f)  recog¬ 
nition.  Another  criterion  area  was  job  satisfaction,  which  was  measured  by 
the  Job  Satisfaction  Questionnaire.  This  guestionnaire  contained  30  items 
which  included  descriptions  of  30  factors,  such  as  Feeling  of  Helpfulness, 
Social  Contact,  Self-Improvement  Opportunities,  and  Job  Security.  The  last 
criterion  was  perceived  productivity  and  was  measured  by  seven  items  contained 
within  the  Perceived  Productivity  Inventory  section.  The  items  measured 
perceived  productivity  in  terms  of  the  work  group's  (a)  guantity  of  work  out¬ 
put,  (b)  guality  of  work  output,  (c)  performance  when  high  priority  work 
arises,  (d)  whether  flow  of  work  to  or  from  the  work  group  is  impaired,  (e) 
freguency  of  crash  projects,  (f)  work  group  output,  and  (o)  comparison  of  work 
group  performance  to  that  of  other  work  groups. 

A  small  validation  study  was  conducted  at  Lackland  AFB  to  refine  the 
instruments.  The  results  were  used  to  delete  items  that  did  not  intercor¬ 
relate  well  with  the  stated  factors  and  to  establish  simple  correlaten 
relationships  between  variables  in  the  situational  environment  and  managerial 
area  with  criteria  items.  The  OAP  previously  described  is  the  result  c-f 
revisions  based  on  data  collected  from  the  Lackland  study. 

Potential  organizations  for  testing  the  OAP  were  identified.  These 
included  the  Army  Research  Institute  for  Behavorial  and  Social  Sciences; 
Headguarters,  Air  University;  and  the  LMPC.  The  revised  versions  of  inven¬ 
tories  were  submitted  to  field  testing  on  a  sample  of  4,786  Air  Fcrce  person¬ 
nel  at  six  bases  representing  six  major  commands.  The  composition  of  the 
sample  was  86  percent  male,  14  percent  female,  63  percent  enlisted  personnel, 
17  percent  officers,  and  18  percent  civilian  personnel.  A  few  of  the  signifi¬ 
cant  differences  found  were  that,  overall,  those  who  were  in  an  organization 
more  than  36  months  scored  higher  on  the  previously  identified  criteria  than 
did  individuals  of  shorter  tenure.  Supervisors  of  larger  numbers  of  people 
also  scored  higher  than  did  other  groups  of  individuals.  A  supervisor  who 
used  work-group  meetings  to  set  goals  and  to  solve  problems  had  workers  who 
were  more  satisfied  and  who  perceived  productivity  and  the  organization's 
climate  to  be  better.  Civilians  reported  higher  job  related  satisfaction  than 
did  officers  and  airmen.  Airmen  perceived  work  group  productivity  to  be  lower 

than  did  officers  and  civilians. 


The  application  of  the  OAP  system  is  Air  Force-wide,  and  its  use  has  high 
impact  potential;  therefore,  continuing  research  and  development  efforts  are 
directed  toward  ensuring  the  validity  and  reliability  of  this  process.  The 
IMDC  conducted  a  study  at  Eglin  AFB,  correlating  workers'  perception  of 
productivity  as  determined  by  the  OAP  and  14  independently  derived  measures  of 
productivity.  Preliminary  analyses  of  the  data  showed  a  very  high  relation 
between  these  two  measures.  Two  reliability  studies  were  accomplished  by  the 
LMDC  Directorate  of  Research  and  Analysis.  A  3-week  test  and  retest  made  a 
factor-by-factor  assessment  on  responses  of  19  personnel  at  the  Academic 
Instructors  School  at  Maxwell  AFB.  Pearson  correlations  on  20  factors  ranged 
between  .51  and  .96,  with  only  two  falling  below  .70. 

Application 

The  application  of  the  OAP  was  immediate  and  widespread.  It  is  now  the 
major  instrument  for  organizational  diagnosis  used  by  management  consultants 
from  LMDC  and  other  agencies  throughout  the  Air  Force.  More  than  140,000 
individuals  have  completed  these  surveys. 

Administration  of  the  survey  is  the  first  step  in  the  consultation  pro¬ 
cess.  The  survey  is  given  to  a  stratified  random  sample  of  the  organization 
to  which  LMDC  has  been  invited.  This  means  that  the  sample  is  so  drawn  that 
it  is  made  up  of  the  same  proportions  of  the  different  levels  as  is  the  entire 
organization.  Within  the  levels,  each  and  every  individual  has  an  eoual 
chance  of  being  included  in  the  sample.  Thus  the  small  sample  is  representa¬ 
tive  of  the  entire  organization,  and  sample  results  may  legitimately  be 
generalized  to  the  entire  organization.  The  results  of  the  survey  are  an 
important  feature  in  the  assessment  of  the  organization.  The  results  are 
handled  in  a  confidential  manner  between  LMDC  and  the  client.  After  approxi¬ 
mately  5  to  6  weeks  for  analysis,  feedback  of  data  is  then  provided  to  comman¬ 
ders  and  supervisors  within  the  organization.  When  specific  problems  are 
encountered,  a  consultant  and  supervisor  develop  a  management  action  plan 
designed  to  resolve  the  problem  at  the  level  of  the  organization.  Within  6  to 
9  months,  the  consulting  team  returns  to  readminister  the  survey  instrument  as 
a  means  to  help  assess  the  impact  of  the  consulting  process. 

The  data  from  each  OAP  administration  effort  are  stored  in  a  cumulative 
data  base  currently  containing  over  100,000  records  for  research  purposes. 
These  data  are  aggregated  by  work-group  codes  developed  for  this  instrument. 
The  data  may  be  recalled  by  demographics  such  as  personnel  category,  age,  sex, 
Air  Force  Specialty  Code,  pay  grade,  time-in-service,  and  educational  level. 
Through  factor  analysis,  the  93  attitudinal  items  are  combined  ir  *\o  factors 
which  cover  job  content,  job  interferences,  and  various  types  of  supervisory 
and  organizational  areas. 

Problems  can  be  more  specifically  and  precisely  defined  through  the  use  of 
the  OAP  system.  Survey  results  point  consultants  toward  potential  problem 
areas  which,  when  resolved,  lead  to  increased  efficiency  in  organizations. 
Some  of  the  benefits  include  enhanced  job  satisfaction,  improved  supervisor/ 
subordinate  relationships,  and  increased  retention  and  productivity.  Repre¬ 
sentative  of  empirically  based  information  obtainable  from  use  of  the  OAP  is 
that  related  to  productivity.  Four  leadership  skills  or  characteristics  have 
been  identified  as  critical  in  achieving  high  productivity  from  subordinates: 
(a)  setting  specific  goals,  (b)  making  responsibilities  clear,  (c)  asking  for 
task  improvement,  and  (d)  emphasizing  job  relevance. 


The  OAP  was  used  as  the  sole  instrument  of  the  organization  assessment  by 
the  Deputy  for  Armament  Systems.  As  a  direct  result  of  administering  the  OAP, 
a  Supervisory  Survey  Feedback  Training  course  was  developed  as  a  joint  endea¬ 
vor  with  LMDC  and  the  Air  Force  Institute  of  Technology.  A  follow-on  survey 
conducted  by  LMOC  indicated  dramatic  changes  in  the  organization.  Communica¬ 
tion  at  every  level  had  improved,  and  with  it,  the  morale  and  satisfaction  of 
personnel.  Problem  areas  had  been  solved  where  it  was  possible  to  do  so  at 
the  Deputate  level;  if  needed,  problems  were  elevated  to  receive  high  level 
attention.  The  Deputate  for  Armament  Systems  increased  its  output  more  than 
16  percent  in  one  year. 

This  totally  new  approach  to  consulting  has  been  lauded  by  industry  and 
schools  such  as  Harvard  Business  School,  Massachusetts  Institute  of  Tech¬ 
nology,  and  Carnegie-Mellon  University.  The  same  work  previously  done  by  20 
people  can  now  be  done  more  efficiently  by  6  to  8  people. 

Payoff 

One  Air  Force  Systems  Command  center  had  allocated  .75  million  dollars  for 
a  contract  with  a  consulting  agency.  When  the  LkDC  consulting  reputation 
reached  this  center,  no  outside  contract  was  let;  LMDC  did  the  consulting  with 
a  savings  of  .75  million  dollars  to  the  Air  Force.  This  center  estimated  that 
two  million  dollars  in  consulting  costs  were  avoided  over  time. 

The  OAP  has  been  successfully  transferred  to  private  industry.  A 
representative  of  a  large  industrial  firm  stated,  "You  have  allowed  us  to 
develop  a  capability  for  Human  Resource  Management  which  is  far  superior  to 
anything  developed  and  used  by  industry.  This  is  the  most  sophisticated  and 
effective  approach  to  management  consultation  I  have  seen." 
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EDUCATION  AND  TRAINING 


The  Department  of  Defense  defines  the  education  and  training  area  of 
people- related  research,  development,  test,  and  evaluation  as  follows: 

"Development  of  educational/training  methods  and  media  for  managing, 
designing,  and  evaluating  new-generation  instructional  systems  for  military 
appl ication." 

The  Air  Force  must  continue  to  develop  technology  that  will  lower  training 
costs  while  actually  improving  training  and  maintaining  personnel  readiness  to 
meet  operational  demands.  Toward  this  end,  new  technology  is  being  developed 
in  the  Educational  and  Training  area  as  indicated  by  the  following  examples. 
First,  critical  training  and  performance  problems  associated  with  advanced  Air 
Force  systems  are  being  answered,  especially  operator  coordination  and  team 
performance  and  training  for  team-operated  command  and  control  systems. 
Second,  the  evaluation,  management,  and  delivery  of  Air  Force  On-the-Job 
Training  through  the  development  and  demonstration  of  an  integrated  training 
system  will  be  undertaken.  This  effort  will  result  in  computer-based  systems 
for  administration,  management,  and  delivery  of  instruction  in  new  technical 
training  applications.  A  final  example  is  a  technology  base  for  training  high 
level  and  guickly  perishable  skills  in  simulated  combat  environments.  This 
base  will  be  provided  through  development  and  application  of  new  instructional 
methods,  technigues,  and  devices  for  use  in  training  and  assessing  aircrew 
skills  at  undergraduate,  upgrade,  and  combat  crew  continuation  training  levels. 

Projects  in  this  category  include: 

1.  Tanker/Transport/Bomber  (TTB)  Specialized  Training 

2.  Operational  Test  and  Evaluation  (OT&E)  Handbook  for  Aircrew  Training 
Devices 

3.  E-3A  Flight  Simulator  Follow-On  Operational  Test  and  Evaluation  (OT&E) 


TANKER /TRANSPORT /BOMBER  SPECIALIZED  TRAINING 


Need 


The  Air  Force  has  an  established  requirement  for  manned  aerospace  weapon 
systems.  The  need  exists  to  maintain  the  Air  Force  capability  to  provide 
high-quality  mission-oriented  flight  training  to  Undergraduate  Pilot  Traininq 
(UPT)  students.  This  capability  must  be  maintained  in  order  to  sustain  an 
adequate  supply  of  skilled  pilots,  thus  contributing  to  the  total  defense 
posture  of  the  United  States.  The  opportunity  to  produce  graduates  better 
trained  in  the  skills  and  procedures  required  for  their  initial  assignments  at 
significantly  lower  operational  costs  dictates  that  a  Specialized  Under¬ 
graduate  Pilot  Training  (SUPT)  program  be  initiated  as  soon  as  possible. 

Extensive  studies  have  shown  SUPT  to  be  a  more  cost-effective  method  for 
providing  an  effective  pilot  training  program  beyond  the  mid-1980's  than  is 
the  present  generalized  or  single-track  program.  In  1972,  a  mission  analysis 
study,  led  by  an  Air  Staff/Major  Command  general  officer  steering  qroup, 
supported  SUPT.  In  1977,  the  Air  Force  Vice  Chief  of  Staff  directed  a  study 
that  concluded  SUPT  would  "provide  a  higher  quality  UPT  graduate  due  not  only 
to  the  added  training  requirements  which  can  be  taught,  but  also  because 
training  is  oriented  toward  a  better  transition  into  the  follow-on  traininq  in 
the  confcat  crew  training  squadron."  To  continue  pilot  training  under  the 
current  generalized  program  would  create  a  numerical  insufficiency  in  T-38 
trainers  in  1986.  SUPT  will  allow  effective  use  of  the  T-38  for  fighter/ 
attack/ reconnaissance  specialized  track  training  beyond  the  year  2000.  In 
June  1980,  the  Secretary  of  the  Air  Force  approved  the  SUPT  concept  for  future 
UPT.  This  approval  justifies  further  planning  and  programming  actions  to 
implement  SUPT. 


R<5cD  Solution 


A  research  and  development  effort  was  initiated  (a)  to  identify  the 
special  training  requirements  which  SUPT  would  entail  for  Tanker/Transport/ 
Bomber  (TTB)  pilots,  (b)  to  develop  a  method  for  qeneralizing  any  given  subset 
of  tasks  selected  for  TTB  training  to  the  entire  domain  of  tasks,  and  (c)  to 
develop  a  methodology  for  use  in  the  measurement  of  aircrew  performance.  This 
effort  was  conducted  by  the  Operations  Training  Division  of  the  Air  Force 
Human  Resources  Laboratory  (AFt-RL)  in  response  to  a  request  from  Headquarters 
Air  Training  Command  (ATC).  This  effort  represents  Phase  I  of  a  three-phased 
cooperative  program  between  AFtRL  and  ATC.  Phase  I  involved  the  identifica¬ 
tion  of  training  requirements  for  a  TTB  pilot  training  course.  During  Phase 

II,  ATC  developed  a  syllabus  to  teach  these  requirements,  and  during  Phase 

III,  this  syllabus  will  be  tested. 

Phase  I  required  the  compilation  of  the  training  requirements  for  B-52, 
KC-135,  C-9,  C-5,  C-141,  and  C-130  aircraft.  This  follow-on  training  prepares 
UPT  graduates  for  the  mission  requirements  of  their  initial  assignment. 
Questionnaires  and  other  data-gathering  instruments  were  developed  to  conduct 
17  different  interviews  with  a  total  of  30  training  staff  and  27  instructor 
personnel  and  to  survey  97  staff,  flight,  academic,  and  simulator  instructors 
and  63  students.  An  inventory  of  training  requirements  was  developed  for  each 
of  the  aircraft  previously  listed.  It  contained  22  functional  task  areas  and 
108  specific  tasks.  The  functional  areas  included  such  tasks  as  takeoff, 
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landing,  trail  formation,  and  crew  coordination.  In  order  to  specify  thos^ 
general  training  functions  that  were  common  for  all  the  aircraft,  a  definition 
of  commonality  was  adopted  which  required  that  a  training  function  be  common 
to  all  aircraft  considered  in  the  study.  Following  such  a  definition,  the 
training  functions  having  the  most  commonality  were  tentatively  identified  as 
(a)  cruise,  (b)  asymmetrical  thrust  flight,  (c)  crew  coordination,  and  (d) 
checklist  usage,  pacing,  and  procedures. 

A  method  was  then  developed  for  estimating  the  benefits  to  be  derived  by 
the  Military  Airlift  Command  (MAC)  and  Strategic  Air  Command  (SAC)  from  SUPT 
training  in  these  task  areas.  Data  were  provided  by  the  answers,  given  by  SAC 
and  MAC  training  personnel,  to  questions  about  what  UPT  graduates  need  to 
know.  Additional  data  were  provided  by  SAC  and  MAC  training  manuals  which 
listed  the  tasks  to  be  taught.  Finally,  information  was  needed  on  such  task 
attributes  as  frequency,  duration,  and  criticality.  Estimates  of  data  in 
these  areas  came  from  the  manuals,  from  the  questionnaires  and  interviews,  and 
from  the  recent  military  experience  of  personnel  conducting  the  study. 

Anecdotal  evidence  was  collected  from  the  C-130  Airlift  School  at  Little 
Rock  AFB,  which  has  actually  been  teaching  students  with  both  generalized  UPT 
and  SUPT  backgrounds  and  has  had  a  chance  to  compare  their  performance.  (The 
generalized  UPT  graduates  are  from  the  Air  Force,  whereas  the  Navy  UPT 
graduates  who  go  to  C-130  school  have  been  through  specialized  training  in 
multi-engine  propeller  aircraft.)  The  four  instructors  who  were  interviewed 
reported  that  the  Navy  student  pilots  are  better  prepared  initially  for 
asymmetric  thrust  procedures,  checklist  pacing,  crew  coordination,  perform¬ 
ance,  and  other  activities  which  concern  multi-engine  operation  in  general. 

The  SAC  and  MAC  interviews,  which  were  much  more  extensive,  produced 
comparable  results.  When  asked  to  list  the  students'  weak  areas  and  the 
training  that  caused  recent  UPT  graduates  the  most  trouble,  instructors 
reported  crew  coordination,  asymmetrical  thrust  and  lack  of  rudder  usage  in 
general,  checklist  usage  pacing,  and  communications.  Landings  were  also 
listed,  with  the  major  problems  caused  by  the  different  landing  pitch 
attitudes  between  T-38  and  the  TTB  aircraft. 


KC-135  personnel  identified  instrument  procedures  as  the  major  problem 
area,  with  the  weakest  proficiency  being  slow/ incomplete  cross-check.  The 
cause  appears  to  be  the  students'  limited  initial  flight  director  system 
background  and  then  later  exposure  to  the  KC-135  FD-109  system  for  the  first 
time  at  the  transition  school.  The  C-141  arid  C-130  training  supervisors  and 
instructors  listed  low-altitude  approach  procedures  as  an  area  that  could  be 
improved.  These  low  approaches,  like  the  KC-135  FD-109  system  problem,  are 
command-unique  since  SAC  aircraft  do  not  fly  the  low-altitude  structure. 
Clearly,  SUPT  could  remedy  some  of  these  problems. 

Application 

Graduates  will  be  more  knowledgeable  in  multi-engine  operations  and  in 
their  ability  to  overcome  asymmetric  thrust  problems  after  participation  in  a 
Specialized  Undergraduate  Pilot  Training  program.  A  particularly  beneficial 
aspect  of  the  dynamic  observer  (teaching  technique  by  which  one  trainee 
observes  the  training  situation  and  the  actions  of  another)  program  is  that 


during  the  early  phases  of  training  the  observer  can  handle  the  communications 
and  then,  after  becoming  more  proficient  in  flying  the  aircraft,  more  knowl¬ 
edgeable  of  procedures,  and  more  experienced  in  radio  operation,  this  student 
can  accomplish  all  three.  (The  transition  schools  have  used  this  method  for 
years.)  Thus,  the  TTB  graduate  will  be  much  better  prepared  in  all  phases  of 
communication.  Realistically,  in  the  T-38  the  pilot  cannot  refer  to  the 
checklist  during  all  situations  or  all  phases  of  flight.  In  the  TTB  trainer 
aircraft  it  will  be  possible  and  realistic  to  use  the  checklist  in  the  manner 
that  SAC  and  MAC  desire.  Therefore,  checklist  usage  and  pacing  will  be 
another  task  in  which  the  entering  student  will  be  better  prepared.  The  TTB 
trainer  aircraft  will  fly  a  standard  multi-engine  traffic  pattern  and  will 
undoubtedly  have  a  landing  attitude  and  other  characteristics  closer  to  the 
larger  MAC  and  SAC  operational  aircraft  than  does  the  T-38.  Consequently,  the 
student  will  be  better  prepared  in  landings  and  patterns  under  instrument 
flight  rules.  Since  the  TTB  trainer  will  have  a  minimum  crew  of  two,  there 
must  be  some  coordination  between  the  two  crewmembers.  The  TTB  graduate  will 
thus  have  a  basic  understanding  of  crew  coordination. 

A  simple  formula  was  developed  and  applied  to  three  phases  of  T-38  train¬ 
ing:  classroom  instruction,  simulator  missions,  and  flight.  For  each  phase, 
a  benefit  ~atio  was  calculated.  This  is  the  ratio  of  the  projected  training 
hours  that  could  be  replaced  with  TTB-specific  training  in  a  dual  track  pro¬ 
gram.  Thus  the  larger  the  value  of  the  benefit  rating  the  more  benefit  there 
would  be  in  using  TTB-specific  hours  to  replace  hours  considered  to  be  irrele¬ 
vant. 

Application  of  the  formula  indicated  that  the  most  likely  benefits  of  SUPT 
training  would  include  significantly  better  preparation  of  pilots  for  the  ini¬ 
tial  phases  of  flying  skill  development  after  assignment  to  SAC  and  MAC.  The 
benefit  to  be  received  is  that  of  being  able  to  set  higher  standards  for  the 
students,  resulting  in  more  experienced  and  proficient  graduates— a  better 
entering  product  for  the  transition  schools.  This,  in  turn,  should  result  in 
more  effective  training  in  the  initial  transition  phases. 

Development  of  an  aircrew  performance  measurement  system  was  approached  ty 
defining  the  scope  of  performance  measurement  in  general,  narrowing  that  scope 
to  the  TTB  case,  specifying  modes  of  measurement,  and  recommending  a  general 
mode  for  each  task  under  consideration.  In  addition,  examples  of  rather  de¬ 
tailed  specifications  were  given  for  an  automated  performance  measurement 
system.  The  similarity  of  the  proposed  measurement  concepts  to  those  being 
implemented  for  the  C-5A  Performance  Measurement  System  was  noted. 

Payoff 

The  results  of  this  R&D  effort  guided  the  ATC  development  of  the  TTB 
syllabus  of  training  for  SUPT.  It  helped  ATC  course  developers  determine 
which  training  tasks  to  include  in  the  TTB  training  track  syllabus.  Ulti¬ 
mately,  all  UPT  Wings  and  all  of  the  flying  MAJCOMS  will  benefit  from  the  ATC 
use  of  this  syllabus. 
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This  work  provides  a  logically  and  empirically  constructed  data  base  on 
which  to  bjild  a  TTB  training  syllabus.  The  approach  developed  a  method  that 
addressed  the  extent  to  which  specific  tasks  can  be  expected  to  transfer  to 
other  tasks  not  covered  by  the  training  program.  This  information  can  be  used 
to  answer  questions  concerning  cross-training  between  TTB  aircraft,  as  well  as 
design  questions  regarding  alternative  configurations  for  a  TTB  trainer  air¬ 
craft  and  supporting  simulator.  A  candidate  performance  measurement  system 
incorporating  work  under  progress  with  the  C-5A  provides  relevant  information 
for  ATC  assessment  of  the  TTB  aircrew  performance. 
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Need 


Present  Air  Force  plans  envision  the  acquisif‘  n  of  njmerous  and  expensive 
aircrew  training  simulators  throjgh  FV90.  Because  their  total  procurement 
cost  is  expected  to  exceed  $3  billion,  it  is  imperative  that  these  simulators 
have  proven  effectiveness  in  training  the  tasks  for  which  they  are  intended. 
To  ensure  that  this  is  properly  accomplished,  Air  Force  directives  require 
that  operational  tests  and  evaluations  be  performed  on  each  system.  As  these 
devices  exist  for  the  purpose  of  training,  the  evaluation  of  trainino  effec¬ 
tiveness  will  constitute  a  significant  portion  of  the  estimate  of  their  mili¬ 
tary  utility.  Prior  to  this  R&D  effort,  there  was  no  standard  methodology  for 
addressing  the  issues  involved  in  the  determination  of  system  training  effec¬ 
tiveness.  This  condition  was  primarily  due  to  the  scope  of  this  multidisci¬ 
plinary  task.  Consequently,  a  dedicated  effort,  using  a  team  of  appropriate 
specialists,  was  necessary  to  provide  a  solution  to  the  problem. 

R&D  Solution 


The  Operations  Training  Division  of  the  Air  Force  Human  Resources 
Laboratory  (AFFRL)  was  responsible  for  this  effort  accomplished  in  response  to 
a  request  from  the  Air  Force  Test  and  Evaluation  Center.  The  approach  con¬ 
sisted  of  a  comprehensive  review  of  the  operational  test  and  evaluation  (0"&E) 
literature  and  official  directives,  site  visits  to  Department  of  Defense  test 
and  evaluation  agencies,  and  technical  discussions  and  interviews  with  knowl¬ 
edgeable  personnel  in  the  military  and  civilian  OT&E  communities.  Relevant 
information  from  the  fields  of  psychology,  training,  and  engineering  were 
included  to  ensure  that  the  handbook  provided  thorough  coverage  of  key  areas, 
such  as  training  effectiveness,  media  utilization,  human  factors  engineer i. eg , 
and  device  reliability  and  maintainability.  To  help  ensure  that  the  handbook 
would  be  of  optimum  utility  to  its  intended  users,  a  series  of  reviews  and 
coordination  meetings  were  held  with  Air  Force  test  and  evaluation  personnel 
from  the  Major  Commands  and  the  Air  Force  Test  and  Evaluation  Center  at  each 
stage  of  development. 

There  are  three  general  functions  with  which  the  aircrew  training  device 
( ATD)  OT&E  test  manager/director  must  be  concerned  and  for  which  coveraoe  in 
the  three  volumes  of  the  handbook  was  deemed  appropriate. 

Volume  1.  Planning  and  Management  is  concerned  with  both  general  arm 
specific  ATD  OT&E  planning  and  management  issues.  It  provides  a  guide  for 
prospective  users  of  the  handbook  and  contains  executive  summaries  of  the 
organization  and  content  of  the  handbook  volumes.  Also  discussed  are  the  ATD 
acquisition  process,  the  types  of  tests  and  evaluations  and  their  objectives, 
and  organization  roles  and  responsibilities.  Further,  Volume  1  covers  the  ATD 
OT&E  planning,  execution,  and  reporting  procedures.  Finally,  ATD  acauisition 
and  event  flow  are  discussed. 

Volume  II.  Operational  Effectiveness  Evaluation  treats  the  assessment  of 
ATD  training  utility  during  OT&E.  This  volume  provides  a  general  tutorial  on 
the  fundamentals  of  ATD  utilization,  evaluation  concepts,  and  evaluation 
approaches.  Specific  training  evaluation  techniques  are  presented.  Finally, 
in  recognition  of  the  fact  that  ultimate  ATD  effectiveness  is  also  very  much  a 
function  of  its  instructor/operator  station  (10S)  design,  Volume  II  provides 
highly  specific  guidance  for  IOS  evaluations. 


Volume  III.  Operational  Suitability  Evaluation  is  concerned  witn 
impact  of  various  operational  suitability  factors  such  as  reliability,  main¬ 
tainability,  availability,  and  logistics  supportability  on  the  overall  usefji- 
ness  of  the  AID.  Volume  III  is  intended  to  provide  the  A TO  OT&F  test  manaaer/ 
director  with  basic  understanding  of  the  suitability  areas  (i.e.,  what  is  done 
and  by  whom)  and  provides  specific  "how  to"  techniaues.  Although  the  techni¬ 
cal  methods  and  procedures  will  be  handled  by  trie  appropriate  engineers  and 
technicians  during  the  conduct  of  an  OT&E  suitability  evaluation,  the  handbook 
gives  overall  guidance  and  direction. 

Appl ication 

The  three-volume  0T5£  handbook  published  by  AFFRL  provides  a  standardized 
guide  for  GT&E  of  ATD  training  effectiveness.  The  utility  of  this  product  has 
not  yet  been  empirically  established,  due  to  its  recent  publication  date. 
Strong  face  validity,  plus  the  fact  that  it  is  the  only  comprehensive,  stan¬ 
dard  guide  available,  has  brought  it  wide  acceptance.  Through  its  use,  a  data 
base  will  be  built  up  and  will  provide  empirical  evidence  for  evaluating  the 
procedures  outlined  in  this  handbook. 

The  Tactical  Air  Warfare  Center  (TAWC)  stated  that  this  handbook  is  the 
most  operationally  useful  AFHRL  R&D  technical  report  published  in  1981  a r  d 
1982.  The  handbook  was  made  possible  by  close  consistent  user  involvement  in 
all  phases  of  the  effort.  Most  program  managers  in  TAWC  have  been  issued 
copies  of  the  handbook  to  perform  OT&E  of  aircrew  training  devices. 

The  Military  Airlift  Command  (MAC)  stated  that  AFHRL  studies  have  served 
as  a  training  resource  for  course  managers  and  as  a  guide  to  developino  qual¬ 
ity  training  programs.  Without  this  help,  they  would  have  required  more 
people  and  a  longer  development  period  and  would  have  developed  a  lower  Qual¬ 
ity  end-product. 

MAC  planning  for  the  Follow-On  Operational  Testing  and  Evaluation  (FOT&E) 
of  a  new  low-level  navigator  visual  simulator  will  be  based  on  the  recommenda¬ 
tions  of  this  handbook.  The  decision  whether  to  purchase  up  to  nine  addi¬ 
tional  visual  systems  with  a  total  cost  of  approximately  $80  million  will  be 
based  on  results  obtained  during  this  FOT&E.  If  these  visual  systems  are 
purchased,  they  will  provide  huge  cost  avoidance. 

Payoff 

MAC  could  not  fully  document  exact  cost  savings  because  their  new  aircrew 
training  programs  were  nor  introduced  until  October  1982.  However,  substan¬ 
tial  savings  are  expected  because  of  reduced  flying  requirements  in  the  actual 
aircraft,  thus  making  that  valuable  time  available  for  other  vital  Air  Force 
requirements. 

This  R&D  product  (a)  provides  Air  Force  leadership  with  direction  in 
making  decisions  for  allocating  huge  expenditures  on  aircrew  simulators  and 
(b)  provides  the  operational  Air  Force  units  consistent  methodology  in  the 
planning  and  conduct  of  aircrew  simulator  test  and  evaluation  programs  for  a 
variety  of  systems.  It  is  expected  that  application  of  this  methodology  will 
result  in  increased  efficiency  and  reliability  in  operational  test  and  eva’ua- 
tion  of  training  equipment.  The  benefits  should  be  cost  savings  in  the  con¬ 
duct  of  the  test  and  improved  reliability  and  validity  in  the  test  findings. 
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E-3A  flight  SIMULATOR  FOLU3W-ON  OPERATIONAL  TEST  AND  EVALUATION 


Need 


The  history  of  operational  training  research  and  development  has  demons 
trated  that  on-site  operationally  oriented  research  is  difficult  when  compared 
to  evaluations  conducted  in  a  laboratory  environment;  however,  the  results 
provide  more  usable  data.  The  most  critical  problem  is  control  of  extraneous 
variables.  Unless  this  problem  is  properly  handled,  the  results  are  confoun¬ 
ded  and  inferences  are  difficult,  if  not  impossible.  Tactical  Air  Command 

(TAC)  did  not  possess  sufficient  in-house  R&D  resources  to  evaluate  the  E-3A 
flight  simulator  capability  to  satisfy  the  flight  crew  simulator's  trainino 
requirements.  R&O  support  from  the  Air  Force  Human  Resources  Laboratory 
(AFHRL)  provided  TAC  with  the  expert  knowledge  and  techniaues  needed  for 
evaluating  the  training  capability  of  the  simulator. 

R&D  Solution 


The  Operations  Training  Division  of  AFHRL  responded  to  the  TAC  request  to 
evaluate  the  simulator's  instructional  features  as  they  relate  to  psychologi¬ 
cal  fidelity  and  training  capability.  AFHRL  behavioral  scientists  developed 
evaluation  questionnaires  and  conducted  standardized  interviews  with  13 
instructor  pilots,  6  instructor  flight  engineers,  and  6  radio  aids  console 
operators.  The  objective  was  to  evaluate  the  compatibility  of  the  flight 
simulator  instructional  features  and  the  consoles  as  compared  with  operator 
functions.  Mack-up  panels  of  the  flight  simulators  were  provided  for  the 
taped  interview  appraisals  which  addressed  human  factor  aspects  and  the 
fidelity  and  training  capability  of  the  simulator.  An  effort  was  made  to 
identify  any  incompatible  features/ characteristics,  their  impact  on  trainino, 
and  possible  improvements.  Data  were  collected  on  both  the  psychological 
fidelity  and  the  training  capabilities  of  the  simulator's  training  features. 

After  statistical  analysis  of  the  data,  AFHRL  scientists  identified 
improvements  for  the  features  that  would  enhance  the  training  capability  and 
improve  training  effectiveness  of  the  simulator.  Recommended  improvements 
were  concerned  with  the  reduction  of  the  number  of  software  switching  opera¬ 
tions  required,  identification  of  and  relocation  of  instructor-operator  dis¬ 
plays  and  controls,  and  replacement  of  the  visual  system  with  a  computer  image 
generating  system. 


Application 

Through  this  effort,  data  were  provided  to  aid  in  identifying  tasks  that 
are  fully  trainable  in  the  simulator  and  creditable  toward  flying  training. 
TAC  reported  that  incorporation  of  the  AFHRL-recommended  changes  in  trainino 
effectiveness  and  operational  effectiveness  significantly  aided  further 
development  and  refinement  of  the  E-3A  mission  crew  training  program.  Addi¬ 
tional  instructional  tasks  are  possible  with  the  improved  Instructor  Operator 
Station. 


Payoff 

Reduction  in  number  of  software  switching  operations  resulted  in  simplifi¬ 
cation  of  operating  procedures  and  minimization  of  opportunities  for  error. 
Recommendations  have  been  incorporated  into  the  three  new,  recently  acquired, 
E-3A  Flight  Simulators.  This  payoff  includes  an  increase  in  flying  safety 
through  optimizing  the  training  potential  of  the  equipment. 


HUMAN  FACTORS 


The  Department  of  Defense  defines  the  human  factors  area  of  people- related 
research,  development,  test,  and  evaluation  as  follows: 

"Development  of  improved  methods  and  technologies  for  the  analysis,  design, 
and  evaluation  of  eauipment/ systems  for  safer  and  more  efficient  operation  and 
maintenance. " 

Continued  inflation  and  constrained  budgets  mean  that  Air  Force  weapon 
systems  must  be  developed,  operated,  and  maintained  with  maximum  efficiency 
and  effectiveness  so  as  to  provide  and  maintain  individual  unit  operational 
readiness  at  the  minimum  life-cycle  cost.  Technology  is  at  the  stage  during 
which  relatively  large  improvements  can  be  obtained  from  modest  costs  applied 
at  the  critical  conceptual  and  design  stages  of  weapon  system  development. 
Current  technical  programs  that  have  outstanding  payoff  potential  include  the 
following:  (a)  demonstration  of  techniques  for  predicting  and  controlling 

manpower  and  training  costs  of  weapon  system  ownership  as  a  function  of  weapon 
system  design  characteristics,  (b)  integration  of  the  many  separate  data  bases 
currently  used  to  track  the  support  requirements  of  major  systems,  and  (c) 
development  of  computer  simulation  models  and  other  analytical  tools  for 
managers  to  use  in  assessing  the  combat  capabilities  of  their  maintenance 
organizations. 

Projects  in  this  category  include: 

1.  Specifications  for  Maintenance  Task  Analysis  and  Logic  Tree 
Troubleshooting  Aids  (LTTAs) 

2.  Format  Options  for  Procurement  of  Technical  Orders 

3.  Maintenance  Resources  Interaction  Models 
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SPECIFICATIONS  FOR  MAINTENANCE  TASK  ANALYSIS 
AND  LOGIC  TREE  TROLBLESHOOT ING  AIDS 


Need 

Significant  progress  has  been  made  in  developing  new  techniojes  foi 
preparation  and  presentation  of  technical  data  for  maintenance.  These 
developments  have  included  improved  maintenance  task  analysis  procedures  for 
developing  the  data  base  and  improved  formats  for  presenting  technical  data, 
Job  Guide  Manuals  (JGMs),  and  Logic  Tree  Troubleshooting  Aids  (LTTAs'1. 
Experience  has  demonstrated  that  JGMs  and  LTTAs  are  superior  to  conventions: 
technical  orders.  It  is  generally  recognized  that  a  comprehensive  maintenance 
task  analysis  is  necessary  for  the  development  of  good  JGMs  and  LTTAs. 
Application  of  these  technologies  has  been  hindered  by  a  lack  of  suitable 
specifications  for  use  in  procuring  technical  data.  A  suitable  specification 
for  the  procurement  of  JGMs  was  available.  However,  no  suitable  specifica¬ 
tions  were  available  for  the  procurement  of  LTTAs  and  for  the  maintenance  task 
identification  and  analysis  (MTI&A)  research  and  development  (R&D).  An  effort 
was  needed  to  develop  specifications  in  these  two  areas. 

R&D  Solution 

This  effort  was  conducted  by  the  Logistics  and  Technical  Training  Division 
of  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  in  response  to  a  reauest 
from  the  Air  Force  Logistics  Command.  A  thorough  review  of  the  state-of- 
the-art  in  performing  the  MTI&A  was  accomplished  to  provide  the  basis  for 
developing  the  draft  specification.  This  was  accomplished  by  first  reviewino 
the  literature  in  the  area  and  then  conducting  extensive  interviews  with 
government  and  industry  personnel  who  are  knowledgeable  of  and  experienced  in 
current  MTI&A  procedures.  Interviews  were  conducted  with  industry  personnel 
who  have  had  experience  in  conducting  research  to  develop  MTI&A  procedures  and 
with  personnel  who  have  had  experience  in  managing  and  conducting  MTI&A 
programs.  Interviews  were  also  conducted  with  Air  Force  and  Army  personnel 
who  have  had  experience  in  procuring  MTI&A  based  technical  data.  The  results 
of  the  literature  review  and  interviews  were  analyzed  and  used  to  provide  the 
basis  for  developing  the  MTI&A  reguirements  included  in  the  specification. 

The  draft  specification  reguires  that  the  contractor  accomplish  specific 
tasks  to  ensure  that  all  reguired  maintenance  tasks  are  identified  and  that 
all  information  required  to  prepare  effective  JGMs  and  LTTAs  is  developed. 
The  tasks  include  the  development  of  (a)  a  task  identification  matrix  to 
identify  all  required  tasks  and  specify  the  level  of  maintenance  (inter¬ 
mediate,  organizational,  depot)  at  which  they  are  to  be  performed,  (b)  a 
description  of  the  intended  user  (abilities,  training,  experience),  (c) 
listings  of  required  support  equipment  and  special  tools,  (d)  guidelines  for 
determining  the  level  of  detail  to  be  included  in  JGMs  and  LTTAs,  (e)  an 
analysis  of  possible  equipment  faults  and  resulting  symptoms,  (f)  effective 
step-by-step  procedures  for  accomplishing  each  task,  and  (g)  action  trees 
outlining  a  troubleshooting  strategy  to  isolate  each  possible  fault.  A 
special  section  is  provided  for  accomplishing  the  MTI&A  for  systems  for  which 
Logistic  Support  Analysis  data  are  available.  The  draft  specification  is 
suitable  for  the  procurement  of  MTI&A  for  organizational  and  intermediate 
levels  of  maintenance.  The  specification  establishes  firm  requirements  for 
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conducting  the  MTI&A  but  allows  the  contractor  considerable  freedom  in  how  the 
analysis  is  accomplished.  The  specification  may  be  used  to  procure  MTI&A  for 
development  of  a  data  base  for  use  in  developing  other  types  of  manuals. 


The  state-of-the-art  in  preparing  and  using  LTTAs  was  reviewed  to  provide 
the  basis  for  developing  the  draft  specification.  This  review  was  accom¬ 
plished  by  searching  available  literature  and  then  conducting  extensive 
interviews  with  government  and  industry  personnel  with  experience  in  develop¬ 
ing,  using,  and  evaluating  LTTAs.  The  results  of  the  interviews  were  analyzed 
to  provide  the  basis  for  establishing  the  requirements  for  developing  LTTAs. 
The  goal  was  to  develop  a  specification  that  establishes  basic  requirements 
for  developing  LTTAs  that  will  result  in  improved  performance  while  still 
providing  enough  flexibility  to  be  suitable  for  application  to  a  wide  variety 
of  applications. 

The  draft  specification  provides  specific  and  general  requirements  for  the 
development  of  LTTAs.  These  include  requirements  for  (a)  task  analysis,  (b) 
development  of  troubleshooting  procedures,  (c)  development  of  checkout 
procedures,  (d)  presentation  formats,  (e)  preliminary  information,  (f)  supple¬ 
mental  and  support  information  (wiring  diagrams,  theory  of  operation,  etc.), 
(g)  varying  levels  of  enrichment,  and  (h)  dual  level  presentation.  The  speci¬ 
fication  is  suitable  for  use  in  procuring  LTTAs  for  use  with  all  types  of 
equipment:  electrical,  electronic,  mechanical,  pneumatic,  hydraulic,  optical, 
and  combinations  thereof.  It  is  suitable  for  procurement  of  LTTAs  for  use  at 
the  intermediate  and  organizational  levels  of  maintenance.  The  specification 
may  also  be  used  for  the  development  of  checkout  and  logic  tree  procedures  for 
use  in  other  types  of  manuals. 

Another  outcome  of  this  effort  was  a  handbook  for  use  in  conjunction  with 
the  draft  MTI&A  specification.  The  handbook  provides  an  overview  of  MTI&A 
processes  followed  by  a  listing  of  fundamental  requirements  to  be  performed 
prior  to  the  actual  start  of  MTI&A.  Once  the  analyst  has  become  knowledgeable 
of  the  preliminary  requirements,  the  handbook  provides  the  direction  on  how  to 
perform  MTI&A. 


Application 

The  major  user  of  this  research  is  the  Air  Force  Logistics  Command,  but 
the  real  user  is  ultimately  the  entire  Air  Force.  Logistics  Command  mainten¬ 
ance  managers  stated  that  information  derived  from  this  effort  is  being 
utilized  to  write  a  standard  on  acquisition  of  technical  orders.  Already  they 
have  noted  increased  aircraft  readiness,  reduced  downtime,  and  reduced  turn¬ 
around  time.  They  estimate  multi-million-dollar  savings  in  reduced  mainten¬ 
ance  costs  and  improved  readiness. 

Specifically,  the  troublsshooting  procedures  which  resulted  from  the  LTTA 
specification  are  enabling  all  levels  of  maintenance  personnel  to  troubleshoot 
more  accurately  and  thus,  to  use  fewer  spare  parts.  The  MTI&A  specification 
is  resulting  in  technical  data  that  are  more  complete,  more  accurate,  and  more 
usable  by  the  technician.  The  result  is  improved  maintenance,  especially  by 
first-term  technicians. 


The  handbook  is  a  major  source  of  material  for  a  short  course  on  technical 
data  management  being  taught  by  the  Air  Force  Institute  of  Technology.  The 
use  of  the  handbook  considerably  strengthens  the  performance  of  a  new  or 
inexperienced  technical  order  manager.  The  use  of  the  handbook  should  signi¬ 
ficantly  improve  the  quality  of  technical  data  procured  for  Air  Force  use. 
The  contents  of  the  hancbook  provide  detailed  guidance  to  the  technical  order 
manager  about  all  aspects  of  technical  order  acquisition,  including  deter¬ 
mining  requirements,  selecting  options,  and  managing  the  procurement  process. 
This  is  resulting  in  technical  orders  that  are  timely,  correct,  and  meet  the 
informational  needs  of  the  users.  Better  maintenance  should  be  the  end  result. 


FORMAT  OPTIONS  FOR  PROCUREMENT  (jr  TECHNICAL  ORDERS 


Need 

Technical  orders  are  publications  outlining  the  procedures  for  operation 
and  maintenance  of  Air  Force  equipment  and  systems.  Such  documents  must  be 
responsive  to  the  changing  needs  of  the  Air  Force  and  to  the  changes  that  are 
continually  occurring  in  procedural  data  techniques. 

In  recent  years,  several  types  of  improved  formats  for  technical  data  for 
maintenance  have  been  developed  and  approved  for  use  in  the  Department  of 
Defense.  Examples  of  the  improved  formats  include  Job  Guide  Manuals,  Logic 
Tree  Troubleshooting  Aids,  Fully  Proceduralized  Job  Performance  Aids,  and 
Functionally  Oriented  Maintenance  Manuals.  Available  research  results  indi¬ 
cate  that  these  formats  have  significant  advantages  over  conventional  techni¬ 
cal  orders.  Further,  the  results  indicate  that  each  type  is  more  appropriate 
for  some  applications  than  for  others. 

Although  the  advantages  of  the  new  formats  have  been  demonstrated,  little 
progress  has  been  made  in  applying  them  to  Air  Force  technical  data  programs. 
This  is  due  in  part  to  the  fact  that  many  technical-data  managers  are  not 
sufficiently  familiar  with  the  new  formats  so  they  can  effectively  select  and 
apply  them.  To  overcome  this  problem,  a  handbook  was  needed  to  provide 
technical  data  managers  with  information  on  available  formats,  guidelines  for 
selecting  the  most  appropriate  format  for  a  given  application,  and  guidance  on 
the  special  problems  involved  in  developing  and  procuring  technical  data  in 
each  format. 


R&D  Solution 


The  Logistics  and  Technical  Training  Division  of  the  Air  Force  Human 
Resources  Laboratory  initiated  a  research  and  development  (R&D)  effort  in 
response  to  a  request  from  Air  Force  Logistics  Command.  The  purpose  of  this 
project  was  to  assemble  information  that  would  be  useful  to  Air  Force 
Technical  Order  Managers  and  other  personnel  concerned  with  acguiring  data  for 
use  in  developing  technical  orders.  /To  that  end,  three  distinct  objectives 
were  established: 

1.  To  identify  technical  order  acquisition  requirements  and  problems, 
along  with  existing  formats  and  guidelines. 

2.  To  determine  the  specific  information,  requirements,  procedures,  and 
guidelines  that  should  be  made  available  to  the  Technical  Order  Manager. 

3.  To  develop  a  handbook  of  data  and  procedures  for  requirements  deter¬ 
mination;  format  selections;  and  acquisition,  development  and  implementation 
of  technical  orders. 

The  approach  to  meeting  these  objectives  was,  first,  to  gather  information 
relating  to  the  needs  of  the  Managers  by  interviewing  technical  order  profes¬ 
sionals  from  both  the  Air  Force  and  private  industry.  This  stage  of  the  work 
also  included  a  review  of  available  formats,  specifications,  and  applicable 
literature.  The  second  phase  involved  analysis  of  these  data,  with  the 
selection  of  candidate  formats  and  guidelines  and  the  development  of  addi¬ 
tional  guidelines.  Finally,  the  text  and  introductory  presentation  materials 
were  prepared. 

an 


Data  were  analyzed  throuqh  screening  and  categorization  as  to  principal 
character  (e.g.,  foimat  versus  media),  and  the  existence  and  avai lability  of 
descriptive  information,  test  data,  and  preparation  guidelines.  Any  format - 
based  technique  that  had  all  three  types  or  data  was  designated  a  potential 
candidate  for  inclusion. 

The  identification  of  specifications  and  guidelines  involved  the  composite 
listing  of  all  potentially  useful  specifications  and  guideline  documents, 
noting  the  document  and  its  most  recent  issue,  the  Services  using  the  speci¬ 
fication,  and  the  equipment  and  manual  types  for  which  it  was  applicable.  The 
research  team  selected  the  most  significant  specifications  from  this  listina. 
From  the  literature  review,  a  comprehensive  bibliography  was  established  which 
included  more  than  130  research  type  publications. 

The  final  step  was  to  determine  which  format-based  techniques  should  be 
recommended  for  inclusion.  The  ground  rules  established  for  this  selection 
began  with  the  techniques  that  were  already  determined  to  be  format-based  and 
to  have  both  descriptive  data  and  test  data  available.  Empirical  test  results 
from  previous  R&D  efforts  by  the  Air  Force,  Army,  Navy  or  industry  were  re¬ 
quired  before  any  format-based  technique,  specification,  or  guideline  was 
included. 


Application 

Utilization  in  the  operational  community  is  one  indicator  of  the  validity 
of  an  R&D  product  such  as  a  handbook.  The  technical  order  handbook  has  met 
with  positive  reception  and  is  widely  used  within  the  logistics  community. 

The  F-16  System  Program  Office  stated  that  the  handbook  assessment  of  the 
F-16  Job  Guide  concept  is  proving  helpful  in  structuring  not  only  the  F-16  Job 
Guide  Technical  Order  content  but  the  overall  F-16  technical  order  verifi¬ 
cation  effort.  This  office  forwarded  the  handbook  to  the  Tactical  Air  Command 
and  to  the  Air  Force  Test  and  Evaluation  Center  for  further  study  and  possible 
implementation. 

The  6310th  Maintenance  and  Supply  Group  stated  that  there  has  been  a 
serious  lack  of  current  information  in  this  vital  area  in  the  Air  Force  and 
that  the  technical  order  handbook  fills  the  current  need  for  a  consolidated 
source  of  quality  information  available  to  Technical  Order  Managers.  Other 
available  documents  do  not  present  information  in  sufficient  detail  to  train 
personnel  adequately  for  dealing  with  the  complexities  and  problems  associated 
with  the  acquisition  and  management  of  technical  orders. 

The  handbook  will  be  used  at  the  Air  Force  Test  and  Evaluation  Center  to 
enhance  the  training  program  for  personnel  involved  in  the  technical  order 
acquisition  and  verification  process.  Additionally,  the  handbook  will  provide 
training  to  personnel  from  the  using  and  supporting  commands  assigned  to  the 
Combined  Test  Force.  The  handbook  will  also  be  beneficial  to  Technical  Order 
Managers  at  the  System  Program  Offices.  The  information  in  the  handbook 
provides  detailed  explanations  and  instructions  for  developing  an  effective 
technical  order  acquisition  and  management  program  including  such  items  as 
improved  statements  of  work  and  verification  plans.  This  improved  awareness 
of  technical  order  acquisition  will  result  in  oreatly  improved,  more 
cost-effective  technical  orders. 


A  statement  from  the  Directorate  of  Integrated  Logistics  Support  of  the 
Deputate  for  Simulators  affirmed  the  usefulness  of  the  technical  order  hand¬ 
book  saying  that  study  of  the  handbook  and  a  2-week  course  at  the  Air  Force 
Institute  of  Technology  (AFIT)  would  bring  a  new  Technical  Order  Manager  to 
the  same  level  of  proficiency  as  1-1/2  years  of  on-the-job  training.  This 
handbook  is  the  primary  source  of  information  for  the  Technical  Order 
Acquisitions  Course  (Systems  10)  being  taught  at  AFIT.  Since  May  1982,  more 
than  75  students  have  completed  this  course.  This  is  a  2-week  on-going 
course,  and  additional  students  will  continue  to  benefit  from  this  manual.' 

Payoff 

Use  of  the  technical  order  handbook  should  significantly  improve  the 
quality  of  technical  data  procured  for  Air  Force  use.  The  contents  of  the 
handbook  will  provide  detailed  guidance  to  the  Technical  Order  Manager  about 
all  aspects  of  technical  order  acquisition,  including  determining  require¬ 
ments,  selecting  options,  and  managing  the  procurement  process.  This  will 
result  in  technical  orders  that  are  timely  and  correct  and  that  meet  the 
informational  needs  of  the  users.  Better  weapon  system  maintenance  should  be 
the  end  result. 
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Need 


The  growing  concern  with  weapon  systems  readiness  issues  has  resulted  in 
greater  efforts  to  forecast  accurately  the  manpower,  support  equipment,  ana 
spare  parts  required  to  support  a  given  level  of  flying  activity.  Determining 
the  cost-effective  mix  of  these  resources  poses  a  continuing  management 
problem.  As  new  aircraft  enter  the  weapons  inventory  and  operating  and  main¬ 
tenance  procedures  change,  there  is  a  recurring  need  for  reliable  estimates  of 
the  spares,  support  equipment,  and  manpower  resources  necessary  to  support  the 
desired  level  of  operational  activity.  Such  estimates  enable  Air  Force  mana¬ 
gers  to  allocate  personnel  and  material  resources  to  new  as  well  as  existing 
organizations,  thus  ensuring  combat  readiness. 

Failure  rates  and  frequency  of  maintenance  were  traditionally  predicted  in 
terms  of  aircraft  flying  hours.  However,  in  many  cases,  field  data  show  a 
poor  correlation  between  these  variables.  Maintenance  appears  to  be  related 
to  more  specific  utilization  variables,  such  as  cycles,  landings,  sorties,  and 
bursts  fired.  In  some  cases,  it  may  be  a  function  of  more  than  one  such 
variable.  Lack  of  information  on  the  appropriate  variable(s)  to  use  as  a  base 
measure  can  introduce  substantial  error  when  predicting  maintenance  action 

rates  on  new  equipment  from  field  experience  on  similar  items  used  in  a 
different  operational  environment. 

The  Air  Force  Human  Resources  Laboratory  (AFFRL)  has  developed  supporting 
techniques  for  use  with  the  Logistics  Composite  Modeling  (LCOM)  to  forecast 
the  maintenance  manpower  requirements  for  new  and  existing  weapon  systems. 
Changes  in  the  spares  and/or  support  equipment  levels  had  been  shown  to 

require  changes  in  the  manpower  forecasts,  and  the  converse  was  also  true. 

Therefore,  an  advancement  in  the  modeling  technology  to  provide  for  the  inter¬ 
active  forecasting  of  these  resources  was  needed.  Research  and  development 
( R&D)  was  carried  out  to  establish  the  interactive  effects  of  manpower, 
spares,  and  support  equipment  for  a  given  weapon  system  in  a  peacetime  en¬ 

vironment.  Further  study  extended  the  simulation  and  modeling  technique  to  an 
initial  wartime  surge  environment. 

R&D  Solution 


The  Logistics  and  Technical  Training  Division  of  AFI-flL  was  responsible  for 
this  effort,  conducted  in  response  to  a  request  from  the  Directorate  of 
Acquisition  Policy  of  the  Air  Force  Systems  Command.  The  F-15  operational 
fighter  weapon  system  was  selected  for  study,  and  the  research  program 
progressed  through  the  following  stages:  (a)  determine  essential  variables, 
(b)  define  available  data  sources,  (c)  develop  experimental  simulation  plan, 
and  (d)  examine  quantitative  effects  of  variations  in  input  variables  on 
weapon  system  performance.  The  major  variables  considered  in  performing  the 
LCOM  simulation  were  weapon  system  definition,  maintenance  requirements,  and 
operations  r 'quirements.  Prime  data  sources  for  this  study  were  Air  Force  and 
contractor  publications,  Air  Force  maintenance  data  reporting,  and  structured 
interviews  with  operations  and  maintenance  personnel.  The  experimental 
simulation  plan  examined  40  performance  measures  as  a  function  of  (a)  three 
variations  in  flying  programs  including  10,  20,  and  30  flyinq  hours  per  air¬ 
craft  per  month,  (b)  two  maintenance  concepts,  including  cannibalization  and 


deferred  maintenance,  and  (c)  three  resource  levels  of  baseline,  average,  ann 
minimum  for  manpower,  spares,  and  support  equipment.  The  operations  and  main¬ 
tenance  performance  measures,  evaluated  in  each  test  structure,  were  categor¬ 
ized  in  operations,  aircraft,  manpower,  shop  repair,  spares  supply,  and 
support  equipment  measures. 

This  R&0  effort  has  shown  that  computer  simulation  of  F-15  operations  and 
maintenance,  enhanced  with  mathematical  modeling  of  performance,  yields  a 
powerful  analytic  tool  for  estimating  resource  needs  of  a  weapon  system.  Of 
the  classes  of  models  examined,  the  following  results  were  indicated  for 
peacetime  environments,  (a)  Operations  and  aircraft  performance  measures  tend 
to  change  predominantly  as  a  function  of  manpower  level  and  utilization  rate. 
(Spares  levels  help  predict  many  measures  in  these  categories  at  specific 
utilization  rates.)  (b)  Maintenance  workload  measures  are  best  predicted  with 
simple  models  that  rely  on  manpower  and  spares  levels.  (c)  Changes  in  air¬ 
craft  utilization  rates  introduce  a  great  deal  of  variance  in  nearly  all 
measures,  and  of  the  models  tested,  those  which  take  into  account  interactions 
among  spares,  manpower,  and  support  equipment  enhance  the  predictive  power  of 
simpler  models  that  exclude  these  interactions,  (d)  Cannibalization  tends  to 
reduce  slightly  the  importance  of  spares  levels  on  personnel  and  shop  repair 
measures.  However,  in  general,  cannibalization  affects  operations  and  air¬ 
craft  only  when  the  flying  schedule  becomes  severe.  It  should  be  noted  that 
while  the  overall  level  of  spares  was  varied  in  these  simulation  experiments, 
an  optimum  distribution  of  stock  within  any  level  was  always  assumed. 
Deficiencies  in  the  allocation  of  specific  parts  would  show  a  greater  impact, 
(e)  While  support  equipment  levels  account  for  some  variation  in  most  mea¬ 
sures,  when  this  variable  is  included,  the  overall  predictive  power  of  a 
mathematical  model  tends  to  be  reduced.  Some  anticipated  improvements  in  the 
models  are  anticipated  which  may  remedy  this  deficit. 

Further  R&D  extended  the  simulation  and  modeling  technique  to  an  initial 
wartime  surge  environment.  LCOM  was  used  to  simulate  30  days  of  surge  flyino 
activity  assuming  a  fully  operational  wing  of  72  F-15s  and  various  resource 
levels.  Three  levels  of  manpower,  four  levels  of  spares,  and  three  levels  of 
support  equipment  were  combined  factorially,  yielding  36  separate  simula¬ 
tions.  The  interactive  effects  of  the  three  resources  were  examined  for  40 
weapon  system  performance  measures.  Two  regression  models  were  then  derived 
for  each  measure,  one  which  included  surge  day  and  resource  types  as  predic¬ 
tive  terms,  and  a  second  which  treated  all  variations  over  days  as  error. 

The  major  sources  of  variance  in  40  performance  measures  were  attributed 
to  spare  parts  supplies  and  day-of-surge  activity.  As  spares  supplies  avail¬ 
able  at  the  beginning  of  simulation  were  increased,  the  period  for  which  the 
heavy  flying  demands  necessary  in  a  wartime  surge  scenario  were  sustained 
increased.  As  resource  levels  were  quickly  exhausted  on  the  early  days  of  the 
surge,  flying  activity  tended  to  deteriorate  rapidly  across  days.  Manpower, 
sipport  equipment,  and  interactions  among  the  three  principal  resources 
accounted  for  smaller  portions  of  variance  than  did  either  spares  levels  or 
days,  but  did  enhance  predictive  capabilities  for  most  LOOM  measures. 

with  respect  to  the  regression  models  that  were  developed  with  simulation 
results,  it  was  shown  that  a  simple  model  which  included  predictive  components 
based  on  manpower,  spares,  and  equipment  generally  did  not  provide  stable  or 


accurate  estimates  of  system  performance.  Variance  percentages  accounted  for 
with  this  first  type  of  model  ranged  from  6.60  percent  to  78.65  percent.  Sub¬ 
stantial  increases  in  predictive  strength  were  obtained  from  a  model  which,  in 
addition  to  components  based  on  the  three  resources,  included  the  linear  and 
quadratic  days  components.  This  model  predicted  from  12. 04  percent  to  99.5 
percent  of  the  variance  in  the  AO  performance  measures.  The  increase  in 
predictive  strength  appears  to  justify  the  added  complexity  of  the  second 
model. 

Finally,  the  second  model  type  yielded  encouraging  results  concerning  the 
application  of  simulation  and  mathematical  modeling  techniques  to  the  surge 
environment.  However,  the  treatment  of  interdependencies  among  performance 
measures  is  an  aspect  of  modeling  which  can  be  improved.  Simulation  results 
showed  strong  relations  among  many  performance  measures,  relations  which  were 
ignored  in  the  regression  models  employed.  A  model  of  performance  change 
which  postulates  a  causal  relation  between  the  measure  of  interest,  other  per¬ 
formance  measures  and  resource  quantities  can  improve  the  current  models.  The 
first  model  was  tested  and  validated.  The  extension  into  a  second  model  has 
not  yet  been  validated,  but  verification  studies  are  ongoing. 

Application 

This  R&D  will  be  helpful  to  all  Air  Force  LCOM  users,  working  with  both 
developing  and  operational  weapon  systems.  The  products  of  this  study  provide 
a  means  of  predicting  the  maintenance  demand  rates  of  aircraft  subsystems. 
The  increased  accuracy  in  predicting  maintenance  demand  rates,  used  in  con¬ 
junction  with  the  newly  developing  support  resource  forecasting  models,  will 
allow  for  more  accurate  estimations  of  the  requirements  for  maintenance  man¬ 
power,  spares,  and  support  equipment. 

Payoff 

These  models  and  equations  provide  a  means  to  determine  resource  require¬ 
ments  for  desired  levels  of  readiness.  They  also  provide  the  means  to  perform 
trade-offs  among  various  mixes  of  manpower,  spares,  and  support  equipment  in 
terms  of  effects  on  dollar  costs  and  on  the  ability  of  a  system  to  meet  peace¬ 
time  readiness  and  wartime  employment  objectives. The  products  also  provide 
insight  into  the  determiners  of  equipment  failure  (design  characteristics, 
environmental  factors,  operations  requirements),  thus  allowing  for  corrective 
actions  to  be  taken  where  maintenance  demands  are  unusually  high  or  otherwise 
unacceptable. 
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SIMULATION  AND  TRAINING  DEVICES 


The  Department  of  Defense  defines  the  simulation  and  training  devices  .n<-.i 
of  people- related  research,  development,  test,  and  evaluation  as  follows: 

"Development  of  cost-effective  training  equipment  and  technology  that 
produce  the  needed  performance  for  operation  and  maintenance  of  military 
systems." 

Improved  simulator  training  technology  will  help  ensure  Air  Force  opera¬ 
tional  readiness  and  effectiveness.  Significant  opportunities  exist  with 
flight  and  maintenance  simulators.  An  example  is  through  the  use  of  the 
Advanced  Simulator  for  Pilot  Training.  A  full-visual  research  flight  simula¬ 
tor,  the  ASPT  is  utilized  to  develop  innovative  methods  for  flight  simulator 
training  of  tactics  in  air-to-air,  air-to-ground,  and  terrain-following  war¬ 
fare.  Improved  flight  simulator  hardware  is  being  developed  to  support  these 
training  requirements.  Also,  the  performance  capabilities  of  flight  simu¬ 
lators  are  being  improved  with  special  emphasis  on  developing  and  demons¬ 
trating  improved  visual  image  generation  and  projection  techniques  to  provide 
more  realistic  visual  displays.  Additionally,  maintenance  traininq  will  be 
made  more  cost  effective  by  the  development  and  evaluation  of  maintenance 
simulators.  These  devices  permit  the  simulation  of  malfunctions  and  allow 
hands-on  maintenance  training  and  troubleshooting  to  take  place  without  tying 
up  or  damaging  expensive  real  hardware. 

Projects  in  this  category  include: 

1.  Simulators  for  Maintenance  Training 

2.  Simulator  Training  Requirements  and  Effectiveness  Study  (STRE5) 

3.  Air  Refueling  Park-Task  Trainer  for  Boom  Operators 

4.  Tactical  Flight  Simulation 
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Need 


The  demand  for  hicjnly  trained  technicians  to  operate  and  maintain  the 
sophisticated  Air  Force  systems  has  increased  at  a  rapid  rate  over  the  past 
decade.  New  aircraft  are  continually  being  developed  for  tactical  and  strate¬ 
gic  missions.  These  aircraft,  which  feature  improved  electronics  and  avion¬ 
ics,  have  created  the  need  for  an  advanced  method  of  training  maintenance 
technicians  to  diagnose  and  correct  system  defects.  Most  training  in  avionics 
maintenance  is  conducted  using  actual  equipment  trainers.  This  equipment  is 
in  short  supply,  is  very  expensive,  and  because  of  the  safety  issue  both  for 
personnel  and  the  equipment,  "hands-on"  experience  with  actual  system  equip¬ 
ment  is  severely  limited.  As  a  training  device,  actual  equipment  does  not 
readily  permit  the  controlled  presentation  of  malfunctions  representative  of 
troubleshooting  problems  occurring  frequently  in  operational  settings.  Less 
expensive  real-time  simulators  do  possess  the  capability  for  troubleshooting 
training  which  incorporates  hands-on  practice  to  increase  troubleshooting 
skill  on  a  sample  of  field-related  maintenance  problems.  In  addition  to 
improved  skills  training,  properly  designed  computer-based  training  simulators 
also  have  the  potential  to  release  more  expensive  actual  equipment  for  opera¬ 
tional  readiness.  To  this  end,  objective  data  are  required  to  determine  the 
conditions  and  alternative  simulation  designs  that  result  in  job-competent 
personnel  for  the  least  cost. 


R&D  Solution 


The  Logistics  and  Technical  Training  Division  of  the  Air  Force  Human 
Resources  Laboratory  (AFFRL)  initiated  a  comprehensive  research  project  to 
develop  promising  areas  of  simulation  technology  as  it  applies  to  Air  Force 
maintenance  training.  This  effort  was  carried  out  at  the  joint  request  of  the 
Engineering  Branch  of  the  Aeronautical  Systems  Division,  the  Office  of 
Training  Technology  of  the  Air  Training  Command,  and  the  Training  Programs 
Division  in  Personnel  Programs  of  Headquarters,  Air  Force.  There  have  been 
several  research  and  development  (R&D)  projects  conducted  within  the  following 
areas:  Fidelity  Requirements,  Troubleshooting  Trainers,  Handbooks  and  Model 
Specifications,  and  Field  Evaluation  of  Simulated  Aircraft  Maintenance 
Trainers. 

Fidelity  Requirements 

To  date,  two  major  simulators  have  been  developed  and  evaluated  to  inves¬ 
tigate  the  impact  of  engineering  fidelity  on  cost  effectiveness  and  training 
efficiency.  The  first  device  was  a  three-dimensional  simulator  of  the  6883 
Automatic  Test  Station.  This  device  simulates  the  converter/ flight  control 
test  station  for  the  F-111D  aircraft.  It  was  designed  to  provide  the  highest 
possible  fidelity  in  terms  of  both  operational  and  physical  fidelity.  A  two- 
dimensional  simulator  has  also  been  developed.  This  was  developed  speci¬ 
fically  for  comparison  with  the  three-dimensional  simulator  so  that  a  number 
of  design  parameters  could  be  explored.  With  these  two  simulators  and  the 
actual  equipment  all  available  at  the  same  time,  in  the  same  course,  and  used 
by  the  same  instructors,  a  unique  opportunity  was  created  to  evaluate  many  of 
the  parameters  that  iirr  ot  both  cost  and  training  effectiveness.  Students 
were  assigned  to  one  of  three  training  devices  in  the  course  in  such 
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a  way  that  the  only  difference  was  in  terms  of  the  trainer  jsed.  Student*: 
entering  the  fonverter/Fl iqht  Control  Systems  resident  course  were  systema¬ 
tically  assigned  to  one  of  the  three  training  devices.  T roubleshootinc 
performance,  oaper-and-penci 1  tests,  and  supervisory  rating  forms  were 
developed  through  extensive  field  interviews  of  subject  matter  experts  and 
were  administered  to  compare  the  training.  The  supervisory  rating  forms  for 
specific  tasks  were  obtained  from  the  field  to  provide  additional  feedback 
related  to  job  performance  at  the  duty  station.  Therefore,  a  comparison  was 
conducted  relative  to  the  training  and  cost  effectiveness  among  the  training 
devices  for  a  specific  sample  of  students  confronted  by  a  specific  set  of 
tasks. 

The  findings  of  this  comparative  study  were  as  follows: 

1.  No  significant  perform, ance  differences  among  the  three  training  alter¬ 
natives  were  obtained. 


2.  Trainees  were  proficient  (average  score  85  percent)  in  procedural  tech 
order  tasks  in  all  three  training  conditions. 

3.  Trainees  performed  at  the  same  level  in  troubleshooting  tasks  in  all 
three  training  conditions. 

4.  The  life-cycle  cost  per  student  hour  for  the  actual  eauipment  was  3.7 
times  greater  than  for  the  low  fidelity  simulator  and  2.6  greater  than  for  the 
high  fidelity  simulator. 

5.  Stjdent  attitudinal  acceptance  of  the  simulators  was  very  positive. 
Instructors'  rank  order  preference  of  the  three  training  devices  was  first, 
actual  eauipment;  second,  high  fidelity  device;  and  third,  low  fidelity  device. 

Troubleshooting  Trainers 

An  effort  is  presently  underway  to  investigate  the  potential  for  inter¬ 
active  computer  graphics  to  provide  genera^  purpose  simulation  capability  for 
maintenance  training  with  an  emphasis  on  practice  in  diagnosing  and  correcting 
faults.  The  objectives  of  this  effort  are  (a)  to  demonstrate  the  feasibility 
of  using  interactive  graphics  simulation  as  a  cost-effective  alternative  to 
Actual  Eauipment  Trainers  (AETs),  (b)  to  investigate  the  training  effective¬ 
ness  of  graphics  simulation,  (c)  to  develop  a  functional  specification  for  a 
low-cost  stand-alone  interactive  graphics  learning  environment,  and  (d)  to 
explore  such  issues  as  color,  fidelity,  and  resolution  requirements.  The 
basic  hardware  configuration  consists  of  a  high  resolution  color  graphics 
terminal  and  video  disk  unit  with  graphic  overlays.  This  R&D  will  produce  a 
graphics  simulation  for  the  6883  Automatic  Test  Station  and  specifications  for 
a  low-cost  device  targeted  for  tri-Service  use.  Furthermore,  the  test  bed  can 
be  considered  as  a  prototype  system  for  establishing  functional  specifications 
for  a  variety  of  training  simulations.  This  prototype  can  be  used  to  develop 
simulations  for  new/different  systems  and  eauipment. 

A  parallel  effort  is  underway  to  investigate  the  potential  of  a  general 
purpose,  microfiche-based  Flight  Simulator  Troubleshooting  Trainer  designed  to 
develop  the  troubleshooting  skills  required  in  the  maintenance  of  flight 
simulators.  Both  the  graphics  simulator  and  the  Flight  Simulator 
Troubleshooting  Trainer  are  designed  to  foster  troubleshooting  and  diaanostic 


'.-rills  in  contrast  to  operating  procedures.  Collectively  these  two  -t  j'l i •  , 

and  others  in  the  program,  are  designed  to  obtain  a  wide  ranoe  of  simulates 
applications  that  can  be  generalized  to  other  maintenance  training  environ¬ 
ments. 


Handbooks  and  Model  Specifications 

An  onqoinq  series  of  investigations  have  been  directed  at  improvino  avail¬ 
able  techniques  for  determining  and  documenting  the  training  needs  associated 
with  new  weapons  systems.  This  has  been  documented  in  a  handbook  detail  inn 
the  decision  logic  involved  in  determining  training  requirements  and  a  model 
specification  to  facilitate  the  documentation  of  those  training  requirements. 
A  parallel  development  resulted  in  a  similar  handbook  and  model  specification 
to  aid  procurement  personnel  in  converting  these  training  requirements  into 
procurement  specifications.  An  effort  currently  underway  is  investigating  the 
logistic  considerations  that  impact  on  the  acquisition,  operation,  and  main¬ 
tenance  of  trainers  and  simulators.  This  effort  will  culminate  in  the  produc¬ 
tion  of  a  series  of  decision  aids  to  facilitate  tradeoff  analysis  and  cost 
minimization  strategies  during  the  early  phases  of  weapon  system  acquisition. 

Application 

Collectively  these  studies  have  demonstrated  that  simulators  for  main¬ 
tenance  training  can  provide  efficient  and  effective  training.  The  simulators 
have  been  shown  to  provide  comparable  training  at  less  cost. 

Another  independent  study  on  the  "Cost-Effectiveness  of  Maintenance 
Simulators  for  Military  Training"  by  Orlansky  and  String  presented  the 
following  results.  The  cost  to  develop  and  fabricate  one  unit  or  a  simulator 
was  less  than  60  percent  of  the  cost  of  its  counterpart  actual  equipment 
trainer  in  7  of  11  cases  investigated.  The  cost  of  fabricating  an  additional 
jnit  of  the  simulator  was  less  than  ?0  percent  of  the  cost  of  its  counterpart 
actual  equipment  trainer  in  9  of  these  11  cases.  Acquisition  and  use  of  a 
maintenance  simulator  over  a  15-year  period  would  cost  38  percent  as  much  as 
an  actual  equipment  trainer,  according  to  the  only  life-cycle  cost  comparison 
that  has  been  reported.  Since  maintenance  simulators  and  actual  equipment 
trainers  are  equally  effective  and  since  maintenance  simulators  cost  less,  it 
is  concluded  that  maintenance  simulators  are  cost  effective  compared  to  actual 
equipment  trainers.  This  finding  is  qualified  because  it  is  based  on  a  limi¬ 
ted  number  of  comparisons,  because  effectiveness  is  based  primarily  on  school 
achievement  rather  than  on-the-job  performance  and  because  it  is  based  primar¬ 
ily  on  acquisition  rather  than  on  life-cycle  costs. 

The  Air  Training  Command  has  successfully  used  AFHRL  prototype  simulators 
to  train  aircraft  technicians  at  Lowry  AFB  without  breaking  or  degrading  the 
operational  equipment.  Simulators  bridge  the  gap  between  the  student's  basic 
military  training  and  "hands-on"  apprentice  experience  on  the  actual  equip¬ 
ment.  It  is  reasonable  to  expect,  but  as  yet  unproven,  that  such  "hands-on" 
experience  at  the  school  will  result  in  less  flightline  training  to  learn 
maintenance  procedures.  Less  training  time  on  the  flightline  means  (a)  less 
overall  training  cost,  (b)  less  chance  of  damage  to  expensive  aerospace 
systems,  and  (c)  increased  operational  readiness. 


The  big  payoff  of  properly  designed  maintenance  training  simulators 
occurs,  however,  in  the  variety  of  training  content  that  the  simulator  can 
provide.  Some  of  these  training  devices  offer  students  a  safe  and  guiet 
environment  in  which  to  practice  what  has  only  been  theory  up  to  this  point  in 
their  maintenance  career.  Simulators  are  capable  of  supporting  training  in 
troubleshooting  by  presenting  realistic  malfunctions  under  controlled  condi¬ 
tions.  In  the  practice  mode,  the  student  performs  maintenance  procedures  in  a 
step-by-step  manner  using  the  appropriate  technical  order  or  job  guide.  The 
number  or  type  of  simulated  malfunctions  can  be  changed  at  will  since  it  is  a 
software-based  system.  A  student  who  errs  can  be  visually  cued  by  the  com¬ 
puter,  and  the  erroneous  actions  can  be  automatically  measured,  tabulated,  and 
presented  to  the  instructors. 

Further,  the  use  of  operational  equipment  to  teach  certain  maintenance 
procedures  requires  large  volumes  of  fuel.  On  the  simulator,  the  system 
response  to  student  inputs  is  identical  to  that  of  the  aircraft  but  no  fuel  is 
required.  Power  requirements  for  the  simulator  are  also  lower.  Although 
operational  equipment  tends  to  be  a  heavy  power  user,  some  maintenance  simu¬ 
lators  operate  from  "household  current." 

The  experience  and  information  gained  from  follow-on  maintenance  simu¬ 
lation  R&D  is  providing  the  basis  for  more  effective  decisions  in  the  develop¬ 
ment  and  procurement  of  simulation  devices  in  lieu  of  actual  equipment  train¬ 
ers.  The  program  has  developed  improved  methods  for  determining  whether  or 
not  the  training  requirements  justify  the  need  for  a  simulator  or  trainer. 
Studies  within  the  Deputates  for  Simulators  and  Engineering  of  the  Aeronauti¬ 
cal  Systems  Division  are  underway  toward  improving  the  methodology  for  defin¬ 
ing  engineering  specifications  and  improving  acquisition  strategies. 


SIMULATOR  TRAINING  REQUIREMENTS  AND  EEFEC1 IVENI  SS  SIH'V 


Need 


Aircrew  training  is  an  expensive  and  time-cons uming  endeavor.  At  one  time 
or  another,  virtually  all  known  training  methods  and  media  have  been  jsed  to 
develop  operationally  ready  aircrews  and  to  maintain  their  skill  levels.  To 
meet  these  training  needs  in  a  cost-effective  manner,  the  military  Services 
have  shown  increased  interest  in  the  use  of  simulators  and  related  training 
devices.  These  training  media,  known  collectively  as  aircrew  training  devices 
(ATDs) ,  include  cockpit  familiarization  and  procedures  trainers,  part-task 
trainers,  operational  flight  trainers,  weapon  systems  trainers,  and  full  mis¬ 
sion  trainers. 

Recent  reauirements  to  economize  on  aircraft  fuel  have  provided  strong 
impetus  for  the  increased  interest  in  ATDs,  but  other  factors  have  contributed 
as  well.  The  other  factors  include  increasingly  congested  airspace,  safety 
during  training,  cost  of  operational  equipment  used  for  training,  and  a  desire 
to  capitalize  on  training  opportunities  that  ATDs  provide  for  training  that 
cannot  be  undertaken  effectively,  safely,  or  economically  in  the  air. 

Because  of  the  advantages  simulation  can  offer  over  other  aircrew  training 
media,  Air  Force  policy  dictates  that  ATDs  will  be  used  to  the  fullest  extent 
to  improve  readiness,  operational  capability,  and  training  efficiency.  Imple¬ 
mentation  of  this  policy  requires  specific  technical  guidance.  Information  on 
which  to  base  that  guidance  is  sparse,  however,  and  the  information  that  does 
exist  is  not  always  available  to  those  who  need  it.  Phase  II  of  the  overall 
Simulator  Training  Requirements  and  Effectiveness  Study  (STRES)  program  was 
conceived  as  a  means  of  identifying,  integrating,  and  making  available  current 
simulator  training  information  necessary  to  support  relevant  Air  Force  poli¬ 
cies.  The  base  of  information  provides  guidance  for  the  enhancement  of  pre¬ 
sent  training  and  focuses  research  and  development  needed  to  enhance  future 
simulation-based  training. 


R&D  Solution 


This  effort  was  carried  out  by  the  Logistics  and  Technical  Training 
Division  of  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  at  the  request  of 
the  Rated  Management  Division  of  the  Directorate  of  Operations  of  the  US  Air 
Force  Headquarters. 

Studies  of  the  design,  use,  life-cycle  costing,  and  worth  of  ATDs  were 
conducted  at  AFI-fiL. 

The  primary  objectives  of  the  overall  STRES  program  were  to  define,  des¬ 
cribe,  collect,  analyze,  and  document  information  bearing  on  four  key  areas: 

1.  Criteria  for  matching  training  requirements  with  ATD  fidelity  features. 

2.  Criteria  for  matching  ATD  instructional  features  with  specific  train¬ 
ing  requirements. 
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3.  Principles  of  effective  and  efficient  utilization  of  ATDs  to  accom¬ 
plish  specific  training  requirements. 

4.  Models  of  factors  influencing  the  life-cycle  cost  and  worth  of  owner¬ 
ship  of  ATDs. 

The  Air  Force  plan  for  accomplishing  these  objectives  involved  a  four- 
phase  effort.  Phase  I  was  concluded  prior  to  the  initiation  of  the  present 
study.  It  was  an  Air  Force  planning  activity  that  defined  and  prioritized  the 
total  effort.  Phase  II  was  a  29-month  study  that  involved  collecting,  inte¬ 
grating,  and  presenting  currently  available  scientific,  technical,  and  opera¬ 
tional  information  applicable  to  specific  aircrew  training  issues.  Phase  II 
also  involved  the  identification  of  research  and  development  (R&D)  efforts 
needed  to  enhance  future  simulator  training.  Phase  III  is  planned  to  be  a 
research  activity  that  will  provide  additional  information  on  important  simu¬ 
lation  and  simulator  training  questions  that,  as  expected,  could  not  be 
answered  based  on  currently  available  data.  Finally,  building  on  Phases  II 
and  III,  Phase  IV  is  planned  as  an  Air  Force  effort  to  integrate  findings, 
publish  relevant  information,  and  provide  for  updating  of  the  knowledge  base 
as  new  information  becomes  available. 

A  tri-Service  Advisory  Team  was  formed  by  the  Air  Force  to  help  guide 
STRE5.  A  principal  task  of  the  Advisory  Team  was  to  participate  in  the 
development  of  objectives  and  guidelines  for  the  conduct  of  the  Phase  II  tech¬ 
nical  effort.  As  a  focus  for  the  effort,  a  set  of  "high  value"  operational 
tasks  was  identified.  The  tasks  selected  were  those  for  which  potential  ATD 
training  benefits  were  judged  to  be  greatest  and  for  which  information  on  ATD 
design,  retrofit,  use,  and  worth  was  believed  to  be  incomplete  or  lacking. 
These  tasks  also  provided  a  focus  for  identifying  questions  and  issues 
reflecting  the  information  needs  of  operational  users  that  were  to  be 
addressed  during  Phase  II.  The  high  value  tasks  identified  by  the  Advisory 
Team  were  (a)  individual  and  formation  takeoff  and  landing,  (b)  close  forma¬ 
tion  flight  and  trail  formation,  both  close  and  extended,  (c)  aerobatics, 
spin,  stall  and  unusual  attitude  recognition,  prevention  and  recovery,  (d) 
low-level  terrain  following,  (e)  air  refueling,  (f)  air-to-air  combat,  both 
guns  and  missiles,  and  (g)  air-to-ground  weapons  delivery. 

Information  from  two  sources  was  used  to  address  Phase  II  program  techni¬ 
cal  objectives.  One  source  was  the  professional  literature,  including  books, 
professional  journals,  and  research  reports.  Approximately  1,100  potentially 
relevant  documents  were  identified.  About  400  of  these  contained  information 
that  was  sufficiently  current,  complete,  and  methodologically  sound  to  be  of 
value  to  program  objectives. 

A  second  information  source  was  the  experience  of  people  intimately 
involved  in  the  design,  use,  and  costing  of  ATDs.  Included  were  ATD  manu¬ 
facturers,  R&D  organizations,  an  airline,  and  operational  ATD  users  in  each  of 
the  military  Services.  A  broad  spectrum  of  ATD  design  and  uses  was  sur¬ 
veyed.  Detailed  interview  guides  were  used. 

Seven  reports  were  prepared  to  document  Phase  II  of  this  effort.  Briefly, 
significant  findings  on  the  four  key  issues  are  outlined  below: 
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Fidelity  Features 

The  first  general  STRES  objective  dealt  with  fidelity.  Fidelity  is  de¬ 
fined  as  the  degree  to  which  cue  and  response  capabilities  in  a  simulator  are 
similar  to  those  in  the  actual  aircraft  so  that  learning  of  specific  tasks  in 
the  device  will  enhance  performance  of  those  tasks  in  the  actual  aircraft. 
Specific  fidelity  issues  addressed  were  visual  and  motion  systems. 

Visual  systems  are  vital  for  training  most  piloting  tasks  in  aircraft 
simulators.  RAD,  as  well  as  operational  experience  with  visually  equipped 

simulators,  indicate  that  training  in  such  devices  does  transfer  to  inflight 
performance  for  the  tasks  of  individual  aircraft  approach  and  landing. 

Platform  motion  systems  are  relatively  common  in  the  modern  trainer  and 
have  been  assumed  to  benefit  training  by  contributing  to  the  realism  of  the 
training  environment.  However,  RAD  findings  indicate  that  pilots  are  often 
unaware  of  whether  platform  motion  systems  have  been  turned  on  or  off. 

Instructional  Support  Features 

The  second  general  STRES  objective  dealt  with  relations  between  instruc¬ 
tional  support  features  and  specific  simulator  training  requirements. 

Instructional  support  features  include  all  of  the  hardware  and  software 

capabilities  of  a  device  that  will  permit  an  instructor  to  control  the 

student's  learning  experience.  A  general  finding  was  that  the  use  of  instruc¬ 
tional  support  features  is  directly  related  to  pilot  skill  level  and  associ¬ 
ated  instructional  requirements  rather  than  to  tr,s  tasks  being  trained. 

Utilization  Principles  for  Aircrew  Training  Devices 

The  third  general  STRES  objective  addressed  efffectiveness  and  efficiency 
of  simulator  use.  Any  training  device,  regardless  of  its  sophistication,  is 
only  one  element  in  the  total  training  system,  and  will  be  only  as  effective 
as  its  utilization  allows  it  to  be.  Thus,  the  ways  in  which  the  training  is 
conducted,  the  selection  of  the  instructors,  and  the  monitoring  and  evaluation 
of  the  training  will  all  impact  on  training  device  effectiveness.  STRES  re¬ 
searchers  found  the  more  effective  simulator  training  programs  were  those  in 
which  instructors  have  been  trained  "to  teach"  and  were  taught  specifically 
how  to  use  these  devices  and  their  instructional  support  features. 

Life-Cycle  Cost 

The  fourth  abjective  of  the  program  was  to  develop  models  of  the  factors 
influencing  the  life-cycle  costs  of  simulators.  The  life-cycle  cost  model 
permits  decision  makers,  through  the  use  of  cost  factors  and  actual  cost  data, 
to  determine  costs  of  training  device  alternatives  throughout  their  usable 
life.  The  model  also  can  be  used  to  determine  the  factors  that  influence 
these  costs. 
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The  Strategic  Air  Command  stated  that  recommendations  from  the  STRES 
reports  have  been  analyzed  for  implementation  on  future  simulator  acquisi¬ 
tions.  These  reports  have  been  used  to  definitize  changes  for  both  existing 
and  future  simulators.  Findings  on  the  proper  location  of  instructor  stations 
were  used  to  develop  the  B-1B  Weapon  System  Trainer.  Several  changes  were 
made  to  instructor  training  programs,  future  trainer  specifications,  monitor 
and  evaluation  concepts.  The  R&O  results  were  used  as  background  information 
in  recent  Air  Staff  meetings/briefings  on  a  future  acquisition  program. 


The  Rated  Management  Division  of  Headouarters,  Air  Force,  found  the  STPFh 
reports  a  "good  reference  document  that  draws  together  all  previous  researcr. 
on  training  devices. . .excellent  primer  for  newly  assigned  personnel."  Aft*r 
reading  the  STRES  executive  summary,  managers  at  HQ  USAF  seriously  Questioned 
the  need  for  platform  motion  on  the  simulator  for  the  B-1B  aircraft  and  ini¬ 
tiated  a  follow-on  program  to  develop  a  conclusive  position  on  platform  motion 
for  large  aircraft  simulators.  If  motion  is  eventually  deleted,  life-cycle 
costs  will  decrease.  A  preliminary  estimate  would  be  an  overall  cost  savings 
of  10  million  dollars  for  deleting  motion  on  the  B-1B  simulator  program. 

The  Military  Airlift  Command  (MAC)  is  using  STRES  as  backup  material  and 
sipportive  data.  The  study  provides  good  source  material  to  be  used  in 
developing  and  acquiring  new  training  devices.  Each  of  the  five  course  mana¬ 
gers  dealing  with  simulation  have  been  issued  a  copy  of  Aircrew  Training 
Devices  Utilization  Report.  This  volume  was  used  as  a  standard  reference 
during  development  of  training  programs  for  effective  use  of  the  new  C-130 
flight  simulator.  Use  of  this  report  significantly  reduced  the  time  required 
for  program  development  and  more  importantly,  gave  users  clear  direction  on 
proven  ways  to  use  simulators.  A  second  report,  Aircrew  Training  Devices 
Fidelity  Features  has  been  used  extensively  to  evaluate  fidelity  features  of 
the  C-130  simulator  and  the  visual  system.  Use  of  the  new  C-130  simulator 
will  reduce  MAC  formal  school  flying  hour  requirements  by  at  least  9,000  hours 
per  year.  Part  of  this  reduction  comes  from  increased  simulator  capabilities, 
but  a  significant  part  will  come  from  proper  application  of  these  capabilities. 

The  future  research  plans  volume  of  the  STRES  report  series  identifies  the 
research  needed  to  fill  in  information  gaps  that  were  anticipated  from  the 
outset  of  the  study.  As  anticipated  before  the  initiation  of  STRES  Phase  II, 
complete,  high  confidence  answers  could  not  be  found  for  some  major  questions 
concerning  ATD  design  and  use.  In  some  instances,  information  concerning  a 
given  question  had  never  been  collected  systematically,  or  the  practical 
utility  of  a  given  training  concept  or  ATD  design  feature  had  never  been 
addressed.  In  other  cases,  prior  research  could  not  be  applied  directly  to 
current  training  questions.  Subsequent  experimentation  and  evaluation  are 
still  required  to  provide  the  empirical  data  needed  for  scientific  documen¬ 
tation.  The  base  of  information  developed  in  Phase  II  provides  guidance  for 
the  enhancement  of  present  training  and  focuses  the  R&D  needed  to  enhance 
future  simulation-based  training. 


Payoff 

This  project  documented  available  information  about  ATDs,  indicating  the 
training  effectiveness,  worth-of-ownership,  advantages,  and  constraints  of 
various  aspects  of  aircrew  simulators  for  specific  high  value  training 
requirements.  Management  and  operational  personnel  now  have  a  single  source 
of  all  simulator  features  R&D. 


55 


AIR  REFUELING  PART-TASK  TRAINER  FOR  BOOM  OPERATORS 


Need 


The  economic  rewards  for  using  specialized  simulators  known  as  part-task 
trainers  to  satisfy  particular  aircrew  proficiency  requirements  are  well 
established,  and  have  been  successfully  demonstrated  by  many  military  and 
commercial  aviation  training  programs.  The  preferred  approach  has  been  to  use 
these  simulators  to  provide  both  initial  skills  training  and  refresher  train¬ 
ing  in  specific  mission  elements.  In  1974,  the  Air  Force  identified  the  need 
for  modern  simulators  to  supplement  the  training  available  through  in-flight 
practice. 

The  Strategic  Air  Command  (SAC)  identified  boom  operator  air  refueling 
training  as  one  area  where  simulation  could  be  effectively  applied.  The  heavy 
commitment  of  the  KC-135A  refueling  aircraft  to  support  Air  Force-wide  opera¬ 
tional  and  pilot  training  sorties  sharply  reduced  its  availability  for  boom 
operator  training.  Thus,  SAC  KC-135A  student  boom  operators  were  hampered  in 
developing  hands-on  air  refueling  skills  because  of  limited  flying  time. 
Training  aids  (films,  slides,  and  mock-ups)  provided  a  means  for  the  student 
to  conceptualize  some  operator  skills,  but  the  aids  did  not  allow  an  instruc¬ 
tor  to  present  and  explain  basic  refueling  cues  and  did  not  give  the  student 
hands-on  experience.  As  a  result ,  a  need  existed  to  provide  the  student  boom 
operators  a  means  for  effective  practice  of  air  refueling  tasks  in  the  safety 
and  efficiency  of  a  ground  environment. 

SAC  identified  this  specific  requirement  in  1974  for  a  state-of-the-art 
refueling  simulator  with  proven  reliability  and  training  capacity.  From  this 
requirement,  an  Air  Force  program  was  begun  to  design,  fabricate,  and  test  a 
Boom  Operator  Part-Task  Trainer  (BOPTT) . 

R&D  Solution 


The  Operations  Training  Division  of  the  Air  Force  Human  Resources 
Laboratory  ( AFHRL)  responded  to  the  SAC  requirement  by  drawing  on  its  in-house 
expertise  to  perform  initial  operational  test  and  evaluation  while  at  the  same 
time  conducting  three  transfer-of-training  studies.  A  preliminary  evaluation 
of  the  training  capabilities  of  the  BOPTT  was  performed  at  Wright-Patterson 
AFB  in  February  1978.  The  purposes  of  this  qualification  operational  test  and 
evaluation  were  to  provide  an  economical  and  early  look  at  the  training  poten¬ 
tial  of  the  BOPTT  and  to  permit  identification  of  correctable  deficiencies. 

The  preliminary  evaluation,  based  on  judgments  of  eight  highly  experienced 
SAC  instructor  boom  operators,  found  that  fully  acceptable  training  was  provi¬ 
ded  by  the  simulator;  that  the  fidelity  of  the  simulator  was  adequate;  and 
instructional  features  were  useful  and  necessary  adjuncts  for  certain  training 
tasks.  Subsequent  transfer-of-training  investigation  demonstrated  the 
validity  of  these  subjective  judgments. 


In  r-  ipletely  satisfy  all  test  and  evaluation  objectives,  SAC  request., 
that  AFh  iddress  four  primary  issues  in  boom  operator  simulator  training  is 
the  operational  environment.  The  overall  goal  of  the  study  was  to  make 
long-range  determinations  of  the  cost  effectiveness  of  simulator  training  of 
boom  operator  skills  in  a  limited  flying  environment.  These  determinations 
were  to  include  consideration  of  undergraduate  training  in  the  Combat  Crew 
Training  Squadron  (CCTS),  training  of  instructor  boom  operator  personnel,  and 
device  usage  to  maintain  the  proficiency  of  experienced  personnel.  Specifi¬ 
cally,  the  four  primary  objectives  of  the  study  were  as  follows: 

1.  Determine  the  transfer  of  training  from  the  BOPTT  to  the  KC-135A  air¬ 
craft  for  the  CCTS  boom  operator  student. 

2.  Evaluate  the  BOPTT  training  effectiveness  for  the  instructor  boom 
operator  in  the  Central  Flight  Instructor  Course  (CFIC)  when  the  BOPTT  serves 
as  the  sole  training  device. 

3.  Investigate  boom  operator  skill  degradation  and  reacquisition  for 
experienced  personnel  by  substituting  the  BOPTT  for  the  KC-135A  aircraft  as 
the  practice  medium. 

A.  Evaluate  the  cost  effectiveness  of  the  BOPTT  in  the  CCTS  program. 

To  address  these  issues  one  transfer-of-training  study  was  dedicated  to 
the  CCTS  area,  one  to  the  CFIC  area,  and  one  to  the  skill  retention  area.  The 
subjects  were  59  students  enrolled  in  the  boom  operator  CCTS  program  at  Castle 
AFB .  Results  of  the  CCTS  research  indicated  that  all  student  groups  trained 
in  the  BOPTT  required  significantly  fewer  air  refueling  attempts  (50  versus 
71)  to  reach  proficiency  in  KC-135A  air  refueling  skills  than  did  students 
trained  by  the  standard  syllabus.  In  boom  operation,  procedures  and  communi¬ 
cations,  all  BOPTT  trained  students  were  superior  or  equal  to  the  students  in 
the  standard  syllabus. 

Results  of  the  evaluation  of  the  BOPTT  training  effectiveness  for  the 
instructor  course  showed  that  instructor  trainees  who  received  all  traininn 
sessions  in  the  BOPTT  demonstrated  proficiency  eaual  to  that  of  instructor 
trainees  who  received  all  training  in  the  KC-135A  aircraft.  This  one-to-one 
training  transfer  ratio  afforded  by  the  direct  substitution  of  BOPTT  training 
for  aircraft  training  was  a  striking  confirmation  of  the  simulator's 
effectiveness. 

Results  of  the  skill  maintenance  research  and  development  (R&D)  revealed 
that  no  measurable  degradation  of  boom  operator  skills  occurred  for  the  two 
test  periods — 60  and  120  days.  This  was  due  to  the  high  competence  level  of 
the  personnel  used  as  subjects.  Consequently,  no  conclusions  could  be  drawn 
concerning  the  effectiveness  of  the  BOPTT  in  maintaining  the  proficiency  of 
highly  skilled  boom  operators.  Evidence  from  the  other  two  studies  would  lead 
to  a  prediction  that  the  device  should  be  effective  in  this  application. 

The  fourth  primary  objective  was  to  evaluate  BOPTT  cost  effectiveness  in 
the  CCTS  program.  There  were  two  difficulties  encountered  in  meeting  this 
objective.  First,  accurate  fixed-costs  information  on  SAC  boom  operator 
training,  sufficiently  detailed  to  satisfy  precise  accounting  procedures,  was 
not  available.  Second,  an  Air  Force  cost  effectiveness  model  sensitive  to 
critical  training  parameters  did  not  exist. 
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Fortunately,  SAC  did  have  cost  data  on  the  overall  expense  of  trainino  a 
CCTS  boom  operator.  These  cost  data  were  broken  down  by  training  phase  and 
identified  the  direct  costs  in  the  flight  phase.  Although  this  information 
was  not  as  "fine-grained"  as  desired,  it  was  complete  enough  to  be  usable  for 
Initial  Operational  Test  and  Evaluation  purposes.  The  second  difficulty  was 
overcome  by  using  a  very  simple  "substitution"  model.  In  this  model,  the 
costs  associated  with  BOPTT  operation,  weighted  by  the  demonstrated  transfer- 
of-training  ratio  for  the  devices,  were  compared  to  official  SAC  program  cost 
figures. 

Using  this  simple  model,  the  analysis  of  the  potential  savings  that  could 
be  realized  through  proper  BOPTT  utilization  in  CCTS  presents  an  extremely 
favorable  outlook.  According  to  SAC  accounts,  costs  to  graduate  a  CCTS 
student  are  $77,007.  Of  this  figure,  $65,043  is  incurred  in  the  flight  train¬ 
ing  phase,  with  $31,419  being  a  direct  expense.  When  control  group  data  are 
considered,  the  mean  number  of  aircraft  refueling  contact  attempts  reguired  by 
CCTS  students  to  reach  proficiency  was  approximately  71.  This  translates  to 
an  average  cost  of  $442.52  per  contact  attempt.  The  eguivalent  cost  in  the 
BOPTT  is  $24.00  per  contact  attempt. 


The  BOPTT  was  shown  to  be  an  efficient  medium  for  training  all  aspects  of 
boom  operator  skills.  The  dat?  indicate  that  the  transfer-of-training  value 
from  the  BOPTT  to  the  aircraft  was  67  percent  in  the  CCTS  program  and  100 
percent  in  the  CFIC  program.  The  cost  savings  potential  of  the  BOPTT  is  most 
impressive.  Using  SAC  figures  as  a  basis  for  calculation,  it  was  estimated 
that  proper  utilization  of  the  device  could  avoid  over  1  million  dollars  in 
CCTS  program  costs  per  year.  This  estimate  does  not  include  the  savings  in 
personnel  costs  (student  time  and  instructors)  that  would  also  accrue.  The 
findings  of  the  AFHRL  study  has  resulted  in  numerous  substantial  changes  in 
the  CCTS  and  CFIC  training  materials  and  programs.  SAC  indicates,  further, 
that  the  AFHRL  study  results  have  been  used  extensively  in  developing  a  KC-10 
boom  operator  training  program  for  the  KC-10  aircraft. 


Payoff 


The  data  may  be  interpreted  to  show  that,  without  suffering  a  loss  in 
student  proficiency,  a  reduction  of  20  aircraft  contact  attempts  may  be  made 
by  directly  substituting  BOPTT  for  the  KC-135A.  Since  performing  a  contact 
attempt  in  the  BOPTT  costs  $418.52  less  than  performing  its  counterpart  action 
in  the  aircraft,  the  total  savings  per  CCTS  student  would  be  $8,370.40.  Based 
on  an  estimated  CCTS  output  of  170  students  per  year,  the  resultant  savings 
would  be  over  1.4  million  dollars  annually. 
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TACTICAL  FLIGHT  SIMULATION 


Need 


The  experience  of  the  National  Aeronautics  and  Space  Administration  In 
successfully  training  crews  to  perform  moon  landings,  which  could  be  practiced 
through  simulation  only,  has  demonstrated  the  value  of  simulating  tasks  that 
are  impractical  or  impossible  to  practice  in  the  real  situation. 

Experiences  of  the  Air  Force  and  Navy  during  the  Vietnam  era  indicated 
that  pilot  performance  in  a  combat  situation  could  be  considerably  improved  by 
providing  the  pilot  with  experience  in  flying  mock  air-to-air  engagements 
against  dissimilar  aircraft  and  against  the  types  of  ground  defenses  that 
would  be  encountered  in  combat. 

This  need  for  realistic  training  in  a  combat  environment  led  directly  to 
the  establishment  of  the  Red  Flag  and  Blue  Flag  exercises  by  the  Tactical  Air 
Command  (TAC)  and  similar  training  exercises  by  other  commands.  While  such 
exercises  are  excellent  test  beds  for  new  tactics  on  a  squadron  or  wing  level, 
the  duration  is  frequently  too  short  for  the  evolution  of  new  tactical 
approaches  or  for  exploration  of  the  relative  effectiveness  of  alternative 
strategies  on  an  individual  pilot  level. 

With  the  use  of  a  high  fidelity,  full  mission  simulator,  such  as  the 
Advanced  Simulator  for  Pilot  Training  (ASPT),  pilots  can  be  exposed  to  a  laroe 
number  of  combat  simulations  of  sufficient  fidelity  in  a  short  period  of  time 
to  provide  optimum  situations  for  development  of  offensive  and  defensive 
skills.  The  use  of  such  simulation  of  the  combat  mission  has  the  potential  of 
developing  the  pilots'  abilities  to  react  to  the  ever-changing  mission  re¬ 
quirements  with  flexibility  and  judgment,  based  on  "experience."  A  study  was 
undertaken  to  determine  whether  the  simulator  could  be  used  in  this  extended 
role. 


R&D  Solution 


In  March  1979,  the  Operations  Training  Division  of  the  Air  Force  Human 
Resources  Laboratory  (AFHRL)  conducted  a  feasibility  study  in  tactical 
scenarios  at  the  request  of  TAC.  Combat  ready  A-10  aircraft  pilots  flew  a 
simulator  mission  in  which  they  were  to  destroy  an  enemy  tank  with  their  30mm 
gun.  The  simulated  combat  ground  environment  employed  a  modified  air  defense 
system  including  anti-aircraft  and  surface-to-air  missiles  to  thwart  the  A- 10 
pilot's  mission.  Each  of  seven  combat  ready  A-10  pilots  flew  20  runs  con¬ 
sisting  of  ingress  to  the  target  area,  attempt  to  destroy  the  tank,  and  egress 
from  the  hostile  environment. 

All  seven  pilots  showed  increased  capacity  to  perform  both  offensively  and 
defensively  after  repeated  training  runs  in  the  simulator.  The  composite 
curve  displayed  a  clear  acquisition  phase  at  the  beginning,  followed  by  a 
leveling  off  during  the  mid-trials  and  then  a  marked  end  spurt  during  the  last 
few  trials.  On  the  basis  of  whether  the  pilot  hit  or  missed  the  target,  sur¬ 
vived,  or  was  destroyed,  a  chi  square  analysis  was  performed.  The  overall 
analysis  showed  statistical  significance  at  the  0.05  level.  This  simply  means 


that  the  pilot's  offensive  scores  (hitting  the  tank)  and  defensive  pmcedjrn- 
(survival)  are  hichly  correlated.  When  the  mornina  performance  was  compare-: 
with  the  afternoon  performance,  there  was  a  significant  increase  in  survival 
and  offensive  capabilities  in  the  afternoon. 

This  being  the  first  attempt  to  use  the  ASPT  (or  i ny  other  simulator)  fci 
the  production  of  a  full  combat  scenario,  the  preliminary  results  are  Quite 
encouraging.  The  visual  system  was  programmed  to  almost  full  capacity  to 
produce  the  battle  environment  in  which  the  pilots  were  free  to  fly  and 
develop  their  own  tactics  in  destroying  the  target  and  surviving  the  threat 
environment.  In  this  sense,  the  pilot  is  operating  in  a  completely  open  loop 
fashion,  as  would  largely  be  done  in  true  combat,  and  as  such,  may  be  able  to 
develop  the  flexibility  and  problem  solving  strategies  pilots  have  to  develop 
in  actual  combat.  The  learning  curves  indicate  the  pilots  are  acquiring  and 
integrating  appropriate  information  so  as  to  improve  their  tactical  per¬ 
formance. 


Application 

It  appears  that  the  simulator  may  be  able  to  supply  an  important  crainina 
methodology  for  tactics  development.  With  the  development  of  greater  computer 
capacity,  the  visual  and  other  systems  may  be  capable  of  producing  dynamic 
changing  combat  environments  that  would  allow  the  combat  ready  pilot  to 
maintain  his  combat  skills  at  a  very  high  level.  Flight  simulation  should 
fill  the  training  void  of  those  flying  tasks  that  are  impractical  or 
impossible  to  practice  in  the  real  aircraft. 

Continued  development  of  combat  environments  with  increasing  variability 
and  flexibility  is  needed  to  test  thoroughly  the  total  concept  of  simulator 
tactics  development.  As  a  follow-on  from  such  development,  comparisons  should 
be  made  between  the  performance  scores  in  exercises  such  as  Red  Flag  with 
pilots  trained  in  simulation  and  with  pilots  who  have  not  had  such  exposure. 
Such  a  program  could  very  well  determine  what  sort  of  transfer  of  training  is 
taking  place  from  the  simulator  to  the  flying  mission. 

Assuming  that  the  seven  pilots  comprised  a  representative  sample  and  that 
the  outcomes  did  indicate  transfer  of  training,  an  analysis  was  done  to  esti¬ 
mate  the  payoff,  in  terms  of  combat  outcomes,  that  might  be  expected  if  pilots 
were  to  receive  realistic  combat  training  in  simulators.  The  tool  for  analy¬ 
sis  was  a  computer  war  game  based  on  a  large-scale,  central  European  combat 
scenario.  The  purpose  of  this  analysis  was  to  examine  how  simulator  training 
might  affect  battle  results  during  the  early,  high-risk  days  of  war — the  first 
A  days — when  pilots  fly  their  first  10  missions  and  are  either  lost  or  survive 
as  combat-experienced  pilots. 


Payoff 

The  learning  curves  derived  from  this  study  were  input  to  the  computer  war 
game.  Analyses  of  the  war  game  results  suggest  that  if  simulator  trainina 
could  maintain  pilots'  combat  skills  at  a  fiveor  six-mission  experience  level, 
the  following  advantages  might  be  achieved  during  the  first  A  days  of  combat: 


60 


1.  60  percent  increase  in  tanks  destroyed  per  aircraft  lost. 


2.  14  percent  reduction  in  total  enemy  ground  strength  (as  compared  to 
results  from  pilots  not  having  simulator  training),  with  overall  enemy  force 
reductions  eguivalent  to  using  45  percent  or  more  Close  Air  Support/ 
Battlefield  Air  Interdiction  sorties. 

3.  20  percent  improvement  in  force  ratio  between  fritrndly  and  enemy 
units,  thus  enhancing  ability  to  influence  campaign  actions  beyond  the  initial 
4  days  of  conflict. 


